+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201020/B1e --output-run DEEP/20201020/B1e/science#step1/20240314T053216Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.1199 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0100 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0110 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/pipeline_orig.qgraph DEEP/20201020/B1e/science#step1/20240314T053216Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 6.0953 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 17.1197 seconds; current memory usage: 0.378 Gibyte, delta: 0.199 Gibyte, peak delta: 0.187 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.378 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0265 seconds; current memory usage: 0.378 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.378 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201020_B1e_science#step1_20240314T053216Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2166 seconds; current memory usage: 0.378 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.378 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.5723 seconds; current memory usage: 0.379 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.379 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 62e22279-bd80-4c2d-ab7e-232fc3edd100 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-14 05:33:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-14 05:33:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 05:33:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-14 05:33:12 proc_lsst.multi:170 [INFO] [multi] job_id 40354 proc_lsst.multi INFO: [multi] job_id 40354 2024-03-14 05:33:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:12 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 40354 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 40354 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 40354 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash c03c6b37366855d36fee4563e54f2a49 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63fb02a1-d5d2-4e79-ad06-7fbd2eb92c1a_isr_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63fb02a1-d5d2-4e79-ad06-7fbd2eb92c1a_isr_947244_57.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 9b63dc54c4c537274b7e4b1944f62e82 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a6edb0d7-8342-4686-9c01-09db2e63b3a2_isr_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a6edb0d7-8342-4686-9c01-09db2e63b3a2_isr_947246_44.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash b244351b853a18d67fa342892eae560c parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a96086a5-1166-4651-8563-bb94e92ad551_isr_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a96086a5-1166-4651-8563-bb94e92ad551_isr_947142_15.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash a261cf77cdabd11f7dcbceb60261a52b parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9fa27c8b-9db4-4e28-ad86-3ad43d46469c_isr_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9fa27c8b-9db4-4e28-ad86-3ad43d46469c_isr_947248_30.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 395fb26389bd9915d1e2958ec7c36085 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/24451149-6716-4474-8af8-72b1548c345f_isr_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/24451149-6716-4474-8af8-72b1548c345f_isr_947144_25.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash fdcfb876cfc1955a7918aa736f4e87f6 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e9908082-3f38-4ab3-87e8-66f3ae9ce237_isr_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e9908082-3f38-4ab3-87e8-66f3ae9ce237_isr_947140_49.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6e85baa8c87ae8ba63ad0fc5c6eaafce parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/fc59b8a2-eb1f-4f77-b14d-f908c67e5430_isr_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/fc59b8a2-eb1f-4f77-b14d-f908c67e5430_isr_947246_36.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 1fe4180a9a3bee6f19e03b2bf9d94708 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3bc0f4ae-7924-4265-84c4-527f1e6dc4fc_isr_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3bc0f4ae-7924-4265-84c4-527f1e6dc4fc_isr_947246_56.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 105158ec45c63b9818d89f8c510db39c parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/40184bcd-a69c-4f9e-993f-cb37107e4b39_isr_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/40184bcd-a69c-4f9e-993f-cb37107e4b39_isr_947144_8.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash c50e2fb00e9bbe6a9435f1af8fcf12dd parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/224bf9b4-15ca-4bf5-8775-13dd0d51f094_isr_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/224bf9b4-15ca-4bf5-8775-13dd0d51f094_isr_947248_17.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 6a3113e0be2f5d8242470445d7401b33 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/620a61bc-5427-4149-8cfa-a709ceed85e5_isr_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/620a61bc-5427-4149-8cfa-a709ceed85e5_isr_947142_14.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash c2401ce20a6236d72713be2f91417e19 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/54977349-b37b-4b90-a700-b3edb6daabb7_isr_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/54977349-b37b-4b90-a700-b3edb6daabb7_isr_947248_37.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash b1932ff96274e6679acaa0bcb7a081a4 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a0ac78ae-9fb5-4bdf-b96b-5bd0ea6ee137_isr_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a0ac78ae-9fb5-4bdf-b96b-5bd0ea6ee137_isr_947140_25.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash c765031b18745bfc2800198a46928df9 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/04703ad5-e5c9-4b14-b3ba-f2744bd00fb7_isr_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/04703ad5-e5c9-4b14-b3ba-f2744bd00fb7_isr_947244_23.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash b79f610408f6985cd058629699c5751c parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/72d77da5-e096-461a-86e2-e3056c03476f_isr_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/72d77da5-e096-461a-86e2-e3056c03476f_isr_947248_18.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash f5eb55f06b2fec74e0944e550d7975ed parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ec9729ef-7b2d-4ef3-9160-3c6996f6908a_isr_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ec9729ef-7b2d-4ef3-9160-3c6996f6908a_isr_947142_33.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 67148fcb746b11d217ee0957295bb631 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/89ebfa08-2e40-4d1f-9139-c68dc61a3f42_isr_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/89ebfa08-2e40-4d1f-9139-c68dc61a3f42_isr_947140_10.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash c2b1478262c01060c10fd2179cfb914f parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47ef208d-2963-4a76-89cb-91a82513b4a3_isr_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47ef208d-2963-4a76-89cb-91a82513b4a3_isr_947248_5.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash a91dcf2c0603c19db04204febc809edf parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2416275a-d508-4484-9b5f-11daf56a4449_isr_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2416275a-d508-4484-9b5f-11daf56a4449_isr_947144_35.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash bd6b6645299108d500d7f5147a65add9 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/47fcee62-5537-43ca-915e-8744e5be6211_isr_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/47fcee62-5537-43ca-915e-8744e5be6211_isr_947144_5.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 220a37f6484c0c43d29d7e2b6c07f3a7 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/7b30d795-3127-4fea-b848-8ed806e67908_isr_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/7b30d795-3127-4fea-b848-8ed806e67908_isr_947246_60.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash f3e9b24d5125cad3392c033640b0d91e parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a0890661-8dfa-48fa-9dd5-628e40ac3887_isr_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a0890661-8dfa-48fa-9dd5-628e40ac3887_isr_947144_18.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash d38bd2b0f7e0537685ae5e7b0f32e2e9 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c2f3559d-5253-4f06-b3f1-fbfed9cf0020_isr_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c2f3559d-5253-4f06-b3f1-fbfed9cf0020_isr_947140_13.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 30b2fa6dcb7ebfca3f02a829c13e5ea5 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3e4a6f7-99c6-4110-917b-5be5ee87e7d8_isr_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3e4a6f7-99c6-4110-917b-5be5ee87e7d8_isr_947244_56.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 927c1e39ec74fe258b291efe7517d4f6 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4477e566-f5de-46b2-84e2-a16c831962e9_isr_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4477e566-f5de-46b2-84e2-a16c831962e9_isr_947244_51.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 555d1859c7c4d0d487b52a877ecb72f1 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/7c587559-5e7e-445e-8c71-0b265507e57c_isr_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/7c587559-5e7e-445e-8c71-0b265507e57c_isr_947140_43.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 0ad3193883b0bcdc8a6b22efe07fa525 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7c0f7f9e-4c2b-4ecb-ba3e-27ba6c154212_isr_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7c0f7f9e-4c2b-4ecb-ba3e-27ba6c154212_isr_947248_48.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash f8ed09dc1a99ed29896d1083613966e6 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/d6ac74e1-d5d7-46bd-8548-f8933cbaba77_isr_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/d6ac74e1-d5d7-46bd-8548-f8933cbaba77_isr_947142_41.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash fe9fc52eef7fa81e0813285aa5a90f40 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/42eb66b8-2feb-4be2-b05d-ddc96bfd9976_isr_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/42eb66b8-2feb-4be2-b05d-ddc96bfd9976_isr_947248_45.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 779ecbdf0b34736fee6cb02a91840db4 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c11c871c-3548-4d0c-b44b-75b7475f1808_isr_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c11c871c-3548-4d0c-b44b-75b7475f1808_isr_947246_15.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash ff76a99034e7022741d9f587f9c32f5b parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/abe4ab44-2748-46df-a77d-73d1f0df9178_isr_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/abe4ab44-2748-46df-a77d-73d1f0df9178_isr_947142_58.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 45f9aaafc9aa4771dd39a36641699141 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b575cbea-fbc0-48b6-8fa3-70c8c20c92c8_isr_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b575cbea-fbc0-48b6-8fa3-70c8c20c92c8_isr_947140_30.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 150ab346d487c65cdab20e45d90fb29a parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/34fc6a6e-3eaf-4007-a94f-4d9199cd4065_isr_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/34fc6a6e-3eaf-4007-a94f-4d9199cd4065_isr_947244_9.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 86fc659236db8d1a73c8433e035a3a5b parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5101d044-ca79-45cc-9026-639e35e06239_isr_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5101d044-ca79-45cc-9026-639e35e06239_isr_947140_57.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash f561672e5b4008c54a7ef396997c1284 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2ca74997-af10-4fcd-9193-f7f85ee55a3e_isr_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2ca74997-af10-4fcd-9193-f7f85ee55a3e_isr_947140_23.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 0f8dcb041fc28025d2033bc42c006282 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/20d9b811-a6ed-4b76-86b3-b9ba15826cfa_isr_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/20d9b811-a6ed-4b76-86b3-b9ba15826cfa_isr_947142_7.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 8a33b67d4ae9b68e96d42c4f25cc985b parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/21469f7e-734f-473e-bc95-82514305d6c1_isr_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/21469f7e-734f-473e-bc95-82514305d6c1_isr_947244_1.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 0145d1be226e6eb70a42019948010fe8 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a947c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/55772afd-eafb-4bb2-ba40-70abc9989315_isr_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/55772afd-eafb-4bb2-ba40-70abc9989315_isr_947144_49.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash d6958aca661423fd284a61ea4446371a parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63da5071-71ac-416d-accc-cfdde6187d39_isr_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63da5071-71ac-416d-accc-cfdde6187d39_isr_947244_48.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 02c8033b71c8b11ac2958de451d7393b parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/0722760b-f4a1-4cbc-9e7e-139af04c72d9_isr_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/0722760b-f4a1-4cbc-9e7e-139af04c72d9_isr_947142_45.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash fe263cac7121f3140d1c8c1133da7cdb parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/9230dea2-fee4-45eb-a428-dfbbd105e953_isr_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/9230dea2-fee4-45eb-a428-dfbbd105e953_isr_947246_33.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 06bc3f6306b65ae779a2a17512c18290 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d8bc9764-e6e4-4b46-8434-96794cd5ee6c_isr_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d8bc9764-e6e4-4b46-8434-96794cd5ee6c_isr_947144_23.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 39cbb6472d45f9ced8052eb0de8a8cb1 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/93dc7252-c908-4333-90da-5ba0d6625f20_isr_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/93dc7252-c908-4333-90da-5ba0d6625f20_isr_947142_37.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 78f39d76499da9312e12719eb4ba7c8d parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e10874e4-8b3f-43ab-9151-948d682a5352_isr_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e10874e4-8b3f-43ab-9151-948d682a5352_isr_947140_50.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 2e2216d39185b853c11ebc49f84d779d parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8a3bb6ca-2dfe-4c46-9bb6-f45ef1da6691_isr_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8a3bb6ca-2dfe-4c46-9bb6-f45ef1da6691_isr_947248_1.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 463d405545311fc64b63340365f7ec5a parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/80ac9a0d-c504-4bdd-8b53-15c3e8e8a1ec_isr_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/80ac9a0d-c504-4bdd-8b53-15c3e8e8a1ec_isr_947142_47.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 1d3c0d4d1d36f9cfeab5797335bd0248 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a963e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5948332e-b177-4108-a2df-acfa044b60a8_isr_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5948332e-b177-4108-a2df-acfa044b60a8_isr_947144_32.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash bef16fe97380bc589a328ddb17313a0a parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/b3d22522-5531-4b87-9c48-bd510b9c4c9c_isr_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/b3d22522-5531-4b87-9c48-bd510b9c4c9c_isr_947248_7.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 392de27de084303111849902cdf9900c parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/87818e9d-5e30-4a2d-af51-7d9d1c529780_isr_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/87818e9d-5e30-4a2d-af51-7d9d1c529780_isr_947142_3.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash fe638c953e21cc456ee50a5c357d58cf parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0527d6ed-3aac-4b82-8c9d-79e240fcad0b_isr_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0527d6ed-3aac-4b82-8c9d-79e240fcad0b_isr_947246_11.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 9f955f18c801602853f3abfc4620faaa parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/22f542b1-f5a2-4093-9255-b12915487c7e_isr_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/22f542b1-f5a2-4093-9255-b12915487c7e_isr_947248_44.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash ed290496e1cde05ea1544739f48e8619 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:12 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a927c3b3-26d3-43e1-b489-23b7be1319ef_isr_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a927c3b3-26d3-43e1-b489-23b7be1319ef_isr_947248_56.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 4a1c79813ab6e5f5df6472c5696a33fa parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/afbd734b-02ed-42ee-a597-26127823418a_isr_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/afbd734b-02ed-42ee-a597-26127823418a_isr_947142_59.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 37bfbe09515f3dfde84ff8aec1464b26 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2f872bdd-de3f-444c-ae56-6b159c163ec0_isr_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2f872bdd-de3f-444c-ae56-6b159c163ec0_isr_947144_51.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash cf7efc556929f66c005227f2441cc469 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1c4b1728-bce9-4551-a1f2-36923eeee69f_isr_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1c4b1728-bce9-4551-a1f2-36923eeee69f_isr_947142_55.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 2ef1dcf6cc1923f8195798d865b71047 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f32e2a0e-cf55-49fd-bc33-293fc36b590b_isr_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f32e2a0e-cf55-49fd-bc33-293fc36b590b_isr_947244_34.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e21c3b88305376e6c7349a7f51901539 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3f94fc11-c0c0-48fa-b8ad-ce0f6d42915e_isr_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3f94fc11-c0c0-48fa-b8ad-ce0f6d42915e_isr_947248_2.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 550eb4c910474a66865d65817b2fcff8 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/9228cd52-24df-4bf6-84b1-4aaaee21616d_isr_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/9228cd52-24df-4bf6-84b1-4aaaee21616d_isr_947142_62.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 69dbcff40d3e56c6d7274bc106c7f5c9 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f681b0b2-1681-4e7e-9fe5-152594088514_isr_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f681b0b2-1681-4e7e-9fe5-152594088514_isr_947144_6.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 2e04472361a3c3336cf4a794e81104fc parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/85cafeea-0456-46a4-b07b-9b9560127c00_isr_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/85cafeea-0456-46a4-b07b-9b9560127c00_isr_947246_26.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 5a402d30bbd1b8a271d714155f4b6d3b parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/99d1a248-2653-4be9-926b-e003e30d69e5_isr_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/99d1a248-2653-4be9-926b-e003e30d69e5_isr_947142_5.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash a2ecc734e13c7082ee84ed22f8605fa2 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c3b95d8-0411-4897-b612-b254b241948c_isr_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c3b95d8-0411-4897-b612-b254b241948c_isr_947142_19.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 1d5b75c3ef10552e8648a8e96904b607 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fb2331ed-ae68-49fe-941a-63b1cfd74129_isr_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fb2331ed-ae68-49fe-941a-63b1cfd74129_isr_947244_52.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 300e7d00cc5ba8796c7755ca60fc37cc parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e4886bd7-592c-4a09-99ad-c138520958e0_isr_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e4886bd7-592c-4a09-99ad-c138520958e0_isr_947244_29.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash b574bff7f6485341eaa7a2203661f4a1 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f385f167-3e9c-401b-9e00-ff38f8b09854_isr_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f385f167-3e9c-401b-9e00-ff38f8b09854_isr_947144_46.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 693ff734678a2b28dfe26a9c513bf35e parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c3bb43e1-8ab0-4e43-8bab-b8269e0bd725_isr_947142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c3bb43e1-8ab0-4e43-8bab-b8269e0bd725_isr_947142_40.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 730547f18c01b9a158e75a329a21a575 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/4bd1659a-b3ac-4e30-a90c-228e37ec07e0_isr_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/4bd1659a-b3ac-4e30-a90c-228e37ec07e0_isr_947140_51.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 7f6c754fb566a127a7b8569dd5e10d5f parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 68 parsl.dataflow.strategy DEBUG: Executor multi has 68 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 16 more blocks parsl.executors.status_handling INFO: Scaling out by 16 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-14 05:33:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-14 05:33:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 05:33:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af648991-5ceb-40dd-9508-64eddaa77a62_isr_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af648991-5ceb-40dd-9508-64eddaa77a62_isr_947142_28.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 01ce09466f83d8431652637570bb6fed parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f802199d-553e-4e23-adbf-9b9fdf1aa81f_isr_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f802199d-553e-4e23-adbf-9b9fdf1aa81f_isr_947246_45.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 110a340d50b4005bee3d8908fa9adcb0 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/89daa40f-c4df-4c52-bc47-caee0aa16ec8_isr_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/89daa40f-c4df-4c52-bc47-caee0aa16ec8_isr_947248_60.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash e849cd0d1a998e389b33785a12ce5df5 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5247d168-3387-4178-93ff-a3fcb5d8b5b3_isr_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5247d168-3387-4178-93ff-a3fcb5d8b5b3_isr_947248_4.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 7d03b5dee94628f38c0a31f84f195577 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/2b090fca-c0c7-499c-add8-f8df959bd4a3_isr_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/2b090fca-c0c7-499c-add8-f8df959bd4a3_isr_947244_2.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash f5cd25b596d7c68bc8e20e5401e2d688 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e8613c21-f5fb-4618-bd8d-f9168a17e893_isr_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e8613c21-f5fb-4618-bd8d-f9168a17e893_isr_947144_54.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash d4edd573fa74b864a5e6cec82025c090 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1052041b-13de-4f35-9bac-22a32765afeb_isr_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1052041b-13de-4f35-9bac-22a32765afeb_isr_947248_61.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 6ae5769a28bee0f940c49cf7b433a188 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/63a1735f-1762-467d-b923-bd7c018ce5a0_isr_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/63a1735f-1762-467d-b923-bd7c018ce5a0_isr_947246_55.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 426c09d4968d07450bf38e6f2518305b parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b823ca40-ba01-4d2f-a970-5d8ef43f4803_isr_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b823ca40-ba01-4d2f-a970-5d8ef43f4803_isr_947144_40.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 557fe07e29f97a5783dae79b63c3e188 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/2ad16c68-2ce6-43ce-9361-2cec7ba44b30_isr_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/2ad16c68-2ce6-43ce-9361-2cec7ba44b30_isr_947248_24.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash dc4073c3a928ac9538e60cbc749ab8e8 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1678de06-fae1-41a5-a51a-cda2391cac87_isr_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1678de06-fae1-41a5-a51a-cda2391cac87_isr_947246_10.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash cc04bdc6655c61e39533088c6ec67156 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/c0d3c63e-6c74-4e9f-b071-9303ac98bc8e_isr_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/c0d3c63e-6c74-4e9f-b071-9303ac98bc8e_isr_947248_49.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash da45158d36dfc9011ad98ce4373b38fc parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/da6ad0ae-14f5-4a0d-9284-7c8f9cf7191b_isr_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/da6ad0ae-14f5-4a0d-9284-7c8f9cf7191b_isr_947248_12.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 4a1126f08051fc3ca5cfb97413e15be3 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f145a902-13cf-4d42-88de-38c30c7abf71_isr_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f145a902-13cf-4d42-88de-38c30c7abf71_isr_947142_35.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash cfefc5726e05cf7d41ba558ececb2055 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/087fc3c3-a757-4181-a6c8-46e84767edb5_isr_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/087fc3c3-a757-4181-a6c8-46e84767edb5_isr_947140_20.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 023f3ae6b69d8904ca20834e097d844d parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f464b33f-fb6b-497a-b6c2-b4f83411d5b6_isr_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f464b33f-fb6b-497a-b6c2-b4f83411d5b6_isr_947244_10.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 919802a1a0f20e84d420920bb8ed12ed parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9b6e56d7-3e01-4640-b77c-97bf8b5c3f9c_isr_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9b6e56d7-3e01-4640-b77c-97bf8b5c3f9c_isr_947248_38.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 0e09312fd2de239b29c9ef5552099702 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e8789020-ebce-482b-9f96-1418c50b4034_isr_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e8789020-ebce-482b-9f96-1418c50b4034_isr_947142_23.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 50f5d59801f95cd0f761f5c9dfc2e403 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/688d89f2-9bd5-4f0b-9427-c16e75e52960_isr_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/688d89f2-9bd5-4f0b-9427-c16e75e52960_isr_947244_47.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 02f62b72df57bf0b698af8cd67b843ae parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/59457284-f812-4122-a814-b34e0cf6b0b2_isr_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/59457284-f812-4122-a814-b34e0cf6b0b2_isr_947142_6.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 1e5ddedd5e0fb6eb845cd76d32100e19 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ee64bfcd-bd59-4df5-948e-7ed2840ce1f6_isr_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ee64bfcd-bd59-4df5-948e-7ed2840ce1f6_isr_947246_30.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 98fdd852787f95a639c78384bec186c1 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/60d6a6e9-5548-4120-bf3b-9b508972d8c4_isr_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/60d6a6e9-5548-4120-bf3b-9b508972d8c4_isr_947144_38.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash a1dd9f6b23fae35ad66fb27fe507e99b parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/92634735-c8ba-48dc-9419-98513b8caadc_isr_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/92634735-c8ba-48dc-9419-98513b8caadc_isr_947248_3.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 47486b3f12e760b1909f19b5ca6995dc parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/417043b5-5127-4ca8-b8f0-28f7ee913491_isr_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/417043b5-5127-4ca8-b8f0-28f7ee913491_isr_947248_58.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash e7b7cd4aa554258b7920db84e417aa00 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/9ea6e8f0-1083-4ab6-b719-d679b313c40d_isr_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/9ea6e8f0-1083-4ab6-b719-d679b313c40d_isr_947244_6.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 855ac2c2053b4fd31d742a38af48a15f parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/cfba7de3-13f4-46e1-ab61-49f905a542a9_isr_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/cfba7de3-13f4-46e1-ab61-49f905a542a9_isr_947140_54.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 13fbd62544e81e9b9e41599a729bf863 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/65d29a7c-2cf1-456b-9bc0-45f8f478f67f_isr_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/65d29a7c-2cf1-456b-9bc0-45f8f478f67f_isr_947244_35.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash dabcdd35bb2591821b1d50a79676f670 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/ae5729af-e8d0-47a6-a0f3-1b08ae309c9b_isr_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/ae5729af-e8d0-47a6-a0f3-1b08ae309c9b_isr_947144_24.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash d42653ed4d819490f3737faf9517ea6c parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a4cff87f-9da1-44ee-99f7-0e7a17ec1262_isr_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a4cff87f-9da1-44ee-99f7-0e7a17ec1262_isr_947140_62.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 5154ccd9242e1b0cb7154a014203cef9 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f0e748f4-2c6b-4f0d-a2c0-a68963d84157_isr_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f0e748f4-2c6b-4f0d-a2c0-a68963d84157_isr_947246_6.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 7bcb0dca734f9cb62d99eb7a7c646b4a parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7de9b47e-4f25-4066-ba95-f0cad65aaec0_isr_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7de9b47e-4f25-4066-ba95-f0cad65aaec0_isr_947244_24.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash f1c975c814aae5d92a8557565b0da52c parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b61d2b7c-23aa-46de-a81e-fe2fca5a02f8_isr_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b61d2b7c-23aa-46de-a81e-fe2fca5a02f8_isr_947144_33.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 385a10382b0b227fcfb7b5de8dff56c7 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4dce623b-264e-4bcd-861e-31d9053ae39c_isr_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4dce623b-264e-4bcd-861e-31d9053ae39c_isr_947244_21.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 07bc59f7668332c800448b48e1cb8477 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7a0b755f-f9b0-4317-bb67-54e3c20d013e_isr_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7a0b755f-f9b0-4317-bb67-54e3c20d013e_isr_947144_27.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash c4a54c0d878d68b52e8b7f794f4bba3d parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/72aaeccb-7c98-4a8b-8acd-0c18a1de95bc_isr_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/72aaeccb-7c98-4a8b-8acd-0c18a1de95bc_isr_947142_61.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash e0af636d97ccc8ef97a26f61cae82139 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/edfff64e-6092-4c37-9fff-566f5257a848_isr_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/edfff64e-6092-4c37-9fff-566f5257a848_isr_947246_39.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash ee0eb6710837e725e9fbf08a21b9d17a parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2dc57334-ce84-49db-98e5-be8d85fe835b_isr_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2dc57334-ce84-49db-98e5-be8d85fe835b_isr_947142_24.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 5544029ab6f31e2bc1a7303406e760c0 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/97c09209-1e20-4389-9268-a6f6df9545d7_isr_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/97c09209-1e20-4389-9268-a6f6df9545d7_isr_947246_3.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash dd20489fca73bbefc257bd1ca0dd715e parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/322388a1-ad6f-407e-bcc0-0795c6c2de4f_isr_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/322388a1-ad6f-407e-bcc0-0795c6c2de4f_isr_947140_7.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash b6d9d76c6d6b143a54459e3cf67104b0 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b513c30c-9ad9-43af-85f8-766fd1e7d4b0_isr_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b513c30c-9ad9-43af-85f8-766fd1e7d4b0_isr_947144_11.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash fce09923053fc081e7910c7a1bd7606a parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c68208e-537a-4702-8800-3e371782a235_isr_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c68208e-537a-4702-8800-3e371782a235_isr_947142_2.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 81ed30d4ee7eaf859a6961ed7016b777 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3be68ef7-646a-4bd6-9d08-ac7edb0373a0_isr_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3be68ef7-646a-4bd6-9d08-ac7edb0373a0_isr_947142_52.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 0b4413a0fbfc15c97784b3760ac50a94 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fee92ddb-ae21-477e-82d7-d5baeb37ae32_isr_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fee92ddb-ae21-477e-82d7-d5baeb37ae32_isr_947142_10.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 80675682b4d30439746cb5258f411310 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0f7e3634-d820-416b-bd96-06f4a7ac7f05_isr_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0f7e3634-d820-416b-bd96-06f4a7ac7f05_isr_947140_39.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash a2323639679d6efe597c1703a1fb8abb parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7bb7cfc6-b326-4a3c-afe2-a6f4d5c47063_isr_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7bb7cfc6-b326-4a3c-afe2-a6f4d5c47063_isr_947144_60.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 2a539a7304d2f464e188fd88ef7423e1 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:13 proc_lsst.multi:170 [INFO] [multi] job_id 17031548 proc_lsst.multi INFO: [multi] job_id 17031548 2024-03-14 05:33:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031548 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031548 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17031548 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-14 05:33:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f8927ca-e770-42c3-8f01-067792a69438_isr_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f8927ca-e770-42c3-8f01-067792a69438_isr_947144_59.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash aa6459b74128a4d5f59c3d270518adfd parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6aa115ea-5eed-411a-98b4-1acc2920b18a_isr_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6aa115ea-5eed-411a-98b4-1acc2920b18a_isr_947244_14.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash e97cb5101e098a68e70529b6c60c5c70 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/14e03d8e-33c8-4bd3-a24b-874d82c6dbdf_isr_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/14e03d8e-33c8-4bd3-a24b-874d82c6dbdf_isr_947246_23.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 553eb671483baad2d4f6a239764eec2f parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f237b5a8-0cbd-41ff-9587-596b2eaab252_isr_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f237b5a8-0cbd-41ff-9587-596b2eaab252_isr_947140_18.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash c20fe70a5f8adfc152f465de2bdbcddc parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a5ce70a9-8c71-482f-b3a7-1593805db8ea_isr_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a5ce70a9-8c71-482f-b3a7-1593805db8ea_isr_947248_28.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 24c01cc0c206e177f6708545f71bf3f2 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a5c42b46-116b-467b-bde3-715d055c368d_isr_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a5c42b46-116b-467b-bde3-715d055c368d_isr_947144_42.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash b6a1c4b90ff0072c75d5216255347930 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/bf529ea2-2e7a-4b43-b3d6-36c8fe6b2856_isr_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/bf529ea2-2e7a-4b43-b3d6-36c8fe6b2856_isr_947144_44.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash bbd66690f5e89b4782e278e896ca016d parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/76af494d-e3bf-42c3-aaee-038b7fdbac68_isr_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/76af494d-e3bf-42c3-aaee-038b7fdbac68_isr_947246_54.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 4a0bad686f4ad0b58f1826ab49cce067 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e73544fb-b7ab-496b-bd2c-a9c60d7fa70f_isr_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e73544fb-b7ab-496b-bd2c-a9c60d7fa70f_isr_947140_3.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 9d420ec71583b8847a7cd1744792d2a1 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2412d4a4-c610-454e-80da-3754f295582c_isr_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2412d4a4-c610-454e-80da-3754f295582c_isr_947140_36.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash f0d0a198571b4fe052c1c46239318593 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b0b7af53-c2e5-4f37-b9da-b290b7b968d2_isr_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b0b7af53-c2e5-4f37-b9da-b290b7b968d2_isr_947140_12.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 311664343d3b491f9962637750d758c2 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6e51a0f1-98a2-4ce8-b1cc-334dd3d2a35e_isr_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6e51a0f1-98a2-4ce8-b1cc-334dd3d2a35e_isr_947244_44.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash b72487c078c732d538b63dac32a86582 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7d592577-0de4-48d0-b3a4-85ad5f00c41d_isr_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7d592577-0de4-48d0-b3a4-85ad5f00c41d_isr_947142_53.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash f804d21385d8882054d7d5e73c7d0ab7 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/975358d6-efa8-453d-b7ab-6f0b0ad3c8c6_isr_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/975358d6-efa8-453d-b7ab-6f0b0ad3c8c6_isr_947142_12.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 160de91fbfa884093049af4c999f8df2 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fdbf973e-3f55-4571-b931-2ad8e24b4f0d_isr_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fdbf973e-3f55-4571-b931-2ad8e24b4f0d_isr_947248_22.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash fcc9ae316526e1115b5e135df0a3e98b parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/fc439d27-c2d6-4497-8cdb-ac8cf20d7de7_isr_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/fc439d27-c2d6-4497-8cdb-ac8cf20d7de7_isr_947144_12.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash a3578a51e200cd6942c427553913aa70 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/355cfcc2-3519-4434-904a-8780db3c314a_isr_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/355cfcc2-3519-4434-904a-8780db3c314a_isr_947244_25.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash d8d9654b8037a1c99b3ff19809c46ad7 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/619c025c-8d10-449e-828d-107b46cfadf7_isr_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/619c025c-8d10-449e-828d-107b46cfadf7_isr_947248_27.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 477fc84967f67ee4272b3cebc5f0b7e0 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/dc343157-53ee-49ec-b778-c6da90b7bb6e_isr_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/dc343157-53ee-49ec-b778-c6da90b7bb6e_isr_947140_5.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 8083c88ef9766c24c5fc78d19eaed11c parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c0f465c9-aac5-4450-986d-987753fd8855_isr_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c0f465c9-aac5-4450-986d-987753fd8855_isr_947246_51.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 842273b9de08e1a8c5caada7e95f4964 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/0ff264af-76c6-4a48-917b-e39a9e23e0c7_isr_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/0ff264af-76c6-4a48-917b-e39a9e23e0c7_isr_947248_19.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 7e284fe1dcd26804ffa56b081a109149 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d3c9794a-1901-4903-be0b-9794dd5a7ac2_isr_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d3c9794a-1901-4903-be0b-9794dd5a7ac2_isr_947248_25.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 116013d54f7da253d9aa1cbd008fd139 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/15f7d346-263e-4443-b06c-f2098c5788f5_isr_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/15f7d346-263e-4443-b06c-f2098c5788f5_isr_947140_60.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash bb2fe7567bda78be60d2ff077c60e14a parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f60cd11-0ba2-4658-8dc7-26c1d127d041_isr_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f60cd11-0ba2-4658-8dc7-26c1d127d041_isr_947140_41.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash d19252940fccfee18b1489d1c82c1266 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/943f8244-48a6-4ad6-9661-bc0dc6989c48_isr_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/943f8244-48a6-4ad6-9661-bc0dc6989c48_isr_947140_1.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 0889add53e0498cd1d85e2e6c18ee13d parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/43fdbf49-cbdd-4b08-8865-dd43c73b8ad2_isr_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/43fdbf49-cbdd-4b08-8865-dd43c73b8ad2_isr_947144_29.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash fb1022736caeb140bd6bd84ac8881bc4 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c8fa4a7c-0f6e-42da-84f9-2784cf56ec23_isr_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c8fa4a7c-0f6e-42da-84f9-2784cf56ec23_isr_947142_1.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 8005b05cc67c1bb3bc704869edbfb34d parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b066496b-4206-4e24-801e-56b5d4ae61db_isr_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b066496b-4206-4e24-801e-56b5d4ae61db_isr_947246_41.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 30fed63b09e1085e9dbf4cab9041b478 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/93f00889-092e-4447-871f-558d37d520e0_isr_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/93f00889-092e-4447-871f-558d37d520e0_isr_947140_37.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 6674ec5433e9da446e00fba8cd128dcb parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fdbafef9-f7c0-4ea4-878b-48d341e325df_isr_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fdbafef9-f7c0-4ea4-878b-48d341e325df_isr_947244_22.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash ae280bf32f6e69f4a6b6411b0ade9c83 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/ab9a82f6-2e0c-418f-884d-a3eeff625e9d_isr_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/ab9a82f6-2e0c-418f-884d-a3eeff625e9d_isr_947244_45.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 408443df585f3d45eec8920b46119081 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/851e00c8-32a8-4a5d-80ba-db9f5c3b883d_isr_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/851e00c8-32a8-4a5d-80ba-db9f5c3b883d_isr_947248_55.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 4c2829117f23b6ba40127785f6143807 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/af15b740-49d4-421d-91af-a3abc9957a34_isr_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/af15b740-49d4-421d-91af-a3abc9957a34_isr_947246_40.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 602c0b729b971a88c2676fb356b6ef4c parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9a113520-2d4e-4d63-9bd2-4b35f8417010_isr_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9a113520-2d4e-4d63-9bd2-4b35f8417010_isr_947248_8.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 6b3c7bbbf3883a963860463cfdc67a9f parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/700ceabc-66d3-40bb-a9c6-46b1f31969fe_isr_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/700ceabc-66d3-40bb-a9c6-46b1f31969fe_isr_947140_31.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash f9f5afd7a5fb60c49f812d8ee0f40c53 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:13 proc_lsst.multi:170 [INFO] [multi] job_id 17031549 proc_lsst.multi INFO: [multi] job_id 17031549 2024-03-14 05:33:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031549 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031549 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17031549 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-14 05:33:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/f5958c32-0d84-4b8f-b241-4ee3e0cbdea2_isr_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/f5958c32-0d84-4b8f-b241-4ee3e0cbdea2_isr_947248_14.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash f0712b107ca36e2a3883727eb0456827 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4cce8c2b-2689-4d3e-92ef-0c5f1e1b215a_isr_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4cce8c2b-2689-4d3e-92ef-0c5f1e1b215a_isr_947248_26.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash adc618dc9028d5dda456491af49930c2 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/abb413bd-6bd9-45a8-8c42-f76eadda4df5_isr_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/abb413bd-6bd9-45a8-8c42-f76eadda4df5_isr_947140_4.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash a0f456f52e30d46e4bfad943dd2e6e76 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/8b241efc-9124-4d9b-baab-8ccec97df670_isr_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/8b241efc-9124-4d9b-baab-8ccec97df670_isr_947140_6.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 3ed6eddb1aab68478bec85ce6849dccd parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/76a2d01a-77c5-484d-94ae-e9aa2bfdf0ee_isr_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/76a2d01a-77c5-484d-94ae-e9aa2bfdf0ee_isr_947248_15.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 11c01d784c843cd318013a0030125326 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0337d906-7b56-4839-8651-3aaa06f3f9b2_isr_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0337d906-7b56-4839-8651-3aaa06f3f9b2_isr_947140_27.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 7889e0aae30f3ce417299567ac185db7 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/61ac4532-f78a-435c-8d3a-8f4a81387083_isr_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/61ac4532-f78a-435c-8d3a-8f4a81387083_isr_947142_34.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash e743039ae89c836a608720aaa0178a9d parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e66e8e96-6f93-4e43-ad0d-fc1e89b5e12d_isr_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e66e8e96-6f93-4e43-ad0d-fc1e89b5e12d_isr_947140_19.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash f1d786293f0a2eee178aae814b85d583 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/910fdbfa-f81a-4f16-9d05-9776d0410ab3_isr_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/910fdbfa-f81a-4f16-9d05-9776d0410ab3_isr_947246_48.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 921099fa068ddb27827335a7735b6587 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/49f16a2a-8e7d-493c-8ede-0a70450e5198_isr_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/49f16a2a-8e7d-493c-8ede-0a70450e5198_isr_947244_28.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash cfc7dddaec82a1b3211dbb8eaf40260d parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/480d881f-fdd9-4324-b6da-c4d55ed0625d_isr_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/480d881f-fdd9-4324-b6da-c4d55ed0625d_isr_947246_13.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash f66892cb736008aa6747b72c5e2c9eac parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e926dadc-1225-43a9-a288-f3ce31ea360f_isr_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e926dadc-1225-43a9-a288-f3ce31ea360f_isr_947140_53.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 8b9f64395163897d2672bf4e877eedaf parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/4481e27f-f58b-4982-9811-f8ced0095776_isr_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/4481e27f-f58b-4982-9811-f8ced0095776_isr_947144_4.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash f99e4590a3fb2d854a2a7f30ea726515 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0abd8e38-4f4d-4722-ba73-7a974157628c_isr_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0abd8e38-4f4d-4722-ba73-7a974157628c_isr_947144_19.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 30e0ba9a3d3b7287744a1b75851e0e66 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/1e244e3e-5f12-4956-9332-9b6283b378e9_isr_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/1e244e3e-5f12-4956-9332-9b6283b378e9_isr_947140_28.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 585b025edbd0c9242577bcbd4d205cc1 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cb903a-d03e-45fc-9830-b6ac54686b90_isr_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cb903a-d03e-45fc-9830-b6ac54686b90_isr_947244_49.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 24779355520b22842acb83df011faa7b parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/e8236567-a0bf-4177-b9ed-dc5c2b7e17c0_isr_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/e8236567-a0bf-4177-b9ed-dc5c2b7e17c0_isr_947246_27.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 54b7b84bb5c57f3f868af8446c64ddbd parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ff0cd60a-2f88-40b8-9171-e4337fb775ba_isr_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ff0cd60a-2f88-40b8-9171-e4337fb775ba_isr_947246_47.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 72070668ffe61ff9d92d07b59fe57e89 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a8b795d8-3d6c-4bc0-8028-eef98309b591_isr_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a8b795d8-3d6c-4bc0-8028-eef98309b591_isr_947248_31.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 7444b7e4059d5326e90c47eea85b10ba parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d9676206-3965-494c-8977-c00d545dc129_isr_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d9676206-3965-494c-8977-c00d545dc129_isr_947144_34.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 7137d8bebc8798266b5ed5cd0b482717 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3f2c16c2-08a1-4108-97e0-6e9cfc956372_isr_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3f2c16c2-08a1-4108-97e0-6e9cfc956372_isr_947244_33.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash b93e65287295b2b73f2f4cb7f3954715 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c4d313f8-de5c-46c6-9e2c-0450fadb2563_isr_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c4d313f8-de5c-46c6-9e2c-0450fadb2563_isr_947142_44.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 584f89fc5707f558c9e7bd25b1ff9b5c parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/caf74f01-24e7-497d-b884-1356f61557aa_isr_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/caf74f01-24e7-497d-b884-1356f61557aa_isr_947244_17.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 86909df2601170c49a43d3d197e0049f parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/6ba62a55-3929-4ce3-96a9-927da04e1c94_isr_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/6ba62a55-3929-4ce3-96a9-927da04e1c94_isr_947246_53.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash b5c213979972ee18ef2d5a15531af820 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/5d99e4ae-ccfc-403e-b225-464c4484b1ce_isr_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/5d99e4ae-ccfc-403e-b225-464c4484b1ce_isr_947246_50.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash d553f6f9a39e8c0594979ca03097a7fb parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1808a4cb-a248-4c3e-869a-70fb3db951e1_isr_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1808a4cb-a248-4c3e-869a-70fb3db951e1_isr_947248_23.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 3185d1a952051ad932561327e1d0758a parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:13 proc_lsst.multi:170 [INFO] [multi] job_id 17031550 proc_lsst.multi INFO: [multi] job_id 17031550 2024-03-14 05:33:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031550 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031550 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17031550 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-14 05:33:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c5792c44-8330-426e-acd1-6bbf349d4038_isr_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c5792c44-8330-426e-acd1-6bbf349d4038_isr_947140_33.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash b09339841e91572281e286b120f17155 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0b19ea5f-6d6c-48d6-b0e5-4a9570fc6534_isr_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0b19ea5f-6d6c-48d6-b0e5-4a9570fc6534_isr_947246_8.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 32bb767b83d0197fb3aedf9a029ac443 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d6cd5981-20b1-4618-9421-c63e233202f4_isr_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d6cd5981-20b1-4618-9421-c63e233202f4_isr_947246_19.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 0f094e6e6ade51215e862337e2e2d973 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/6bd83b7b-dbf7-494a-8c53-48cf7c480983_isr_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/6bd83b7b-dbf7-494a-8c53-48cf7c480983_isr_947248_40.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash c1fd0cf05d69636179aa65f9a0bb5ed7 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/71f5032f-67da-4fc9-9647-77022894e589_isr_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/71f5032f-67da-4fc9-9647-77022894e589_isr_947246_21.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 1459e3d5fb34844c099f35c6bc5c7675 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3e261ad8-1f21-47f4-9208-75f6eb888443_isr_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3e261ad8-1f21-47f4-9208-75f6eb888443_isr_947248_10.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 3980ec3f6bbbbd3fb2c1a1942c83ca6d parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/15fcc877-943f-4888-b5f1-619d1291cfb3_isr_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/15fcc877-943f-4888-b5f1-619d1291cfb3_isr_947144_17.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash d4974dc1c131066a9d42a9d1b21825ff parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45b5b37-71ed-4f99-9b04-23bcde05b6fa_isr_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45b5b37-71ed-4f99-9b04-23bcde05b6fa_isr_947244_7.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 4ab4742e5c1c43b7334ba7334daedb9a parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1e89c651-1727-482e-ac41-98a7aa69c287_isr_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1e89c651-1727-482e-ac41-98a7aa69c287_isr_947246_34.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 57fdc95f87e9d5e55792fbeb106d07c7 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2da539c8-8195-4282-ab98-d70929f36fe8_isr_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2da539c8-8195-4282-ab98-d70929f36fe8_isr_947246_35.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 4c93ee0a1fe7d54fb450e8e9d8eb68c8 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/35e9ace6-4daf-48fe-bc42-b50d3ff06c07_isr_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/35e9ace6-4daf-48fe-bc42-b50d3ff06c07_isr_947144_16.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 821c6a768b6b825830d52d04c96f3f6b parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/144085b9-ed42-4c6b-afc4-4d1bc351c0a9_isr_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/144085b9-ed42-4c6b-afc4-4d1bc351c0a9_isr_947144_41.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash ea69d418aea892a3b0f7329c396933c9 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/81b8a874-4ef3-433f-8aeb-3387432541f0_isr_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/81b8a874-4ef3-433f-8aeb-3387432541f0_isr_947248_59.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash cda359439de8f18e8c06dffcb09538ca parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e65b7015-0c73-423a-a14c-e5eac8c2d9ff_isr_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e65b7015-0c73-423a-a14c-e5eac8c2d9ff_isr_947244_26.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash b40778b09bcb4d156f7b475b24179eba parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/92d4da0f-336c-450c-818a-370f68e0738b_isr_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/92d4da0f-336c-450c-818a-370f68e0738b_isr_947142_8.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 216f0582b76cdf49fd58892edde542f1 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7b333195-ba61-4bf7-a492-c02e7699cc3b_isr_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7b333195-ba61-4bf7-a492-c02e7699cc3b_isr_947248_21.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash ac66915b978c456f60f2fb442d311811 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/aa0a3bd7-d1ab-45cf-805c-477834424571_isr_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/aa0a3bd7-d1ab-45cf-805c-477834424571_isr_947246_62.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 640091826e18a830b807cb0ef1415f9c parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3e0f4186-de31-4905-ae37-f59ccfb9abdc_isr_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3e0f4186-de31-4905-ae37-f59ccfb9abdc_isr_947246_2.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash a4938be04d0f6269e836d8ff13263f96 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/11b89a65-4aa8-4eef-ac03-ce89f9d98c9b_isr_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/11b89a65-4aa8-4eef-ac03-ce89f9d98c9b_isr_947248_43.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash ae781dddba28c9d733326f37eb447efc parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/866fa2e5-482d-4616-8a23-0f7578312b4c_isr_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/866fa2e5-482d-4616-8a23-0f7578312b4c_isr_947142_9.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash bda8ddb3c2b3ff038a87b6a504769811 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5bc1c319-bfd4-490b-b4c2-ca64a1f6ebae_isr_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5bc1c319-bfd4-490b-b4c2-ca64a1f6ebae_isr_947142_56.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash dc0d2a760e35090103bc782853762be5 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0f1e8e4c-4589-42ae-ae70-d47bd5dfd6af_isr_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0f1e8e4c-4589-42ae-ae70-d47bd5dfd6af_isr_947144_62.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 4f7b9b1145e50bd428b2fad05447578c parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/20680e9b-bc75-4529-9785-830bcc4352d5_isr_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/20680e9b-bc75-4529-9785-830bcc4352d5_isr_947246_46.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash d4b6c5c7d345372d0482ead993fc46e0 parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d804a5a9-11b6-4021-907c-41eb9470e2f2_isr_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d804a5a9-11b6-4021-907c-41eb9470e2f2_isr_947246_58.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash d2ca00914e65e494789b7153b8af70de parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fbaa49c0-dc80-421a-bbc3-8af3ada00b21_isr_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fbaa49c0-dc80-421a-bbc3-8af3ada00b21_isr_947142_60.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 77a766288641926a19f0af41d9257055 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9a502fb9-1675-4981-ba0f-5d2518fca806_isr_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9a502fb9-1675-4981-ba0f-5d2518fca806_isr_947144_39.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 34df103259d365df8442e4099ac2918b parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ca5746c4-51bc-451a-bf83-6d823298eb3a_isr_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ca5746c4-51bc-451a-bf83-6d823298eb3a_isr_947246_52.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 5ea548e55b066637ca96b3b70e8bfe5a parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8ffc0d7f-bb15-4891-99f9-e0f1c9aaa795_isr_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8ffc0d7f-bb15-4891-99f9-e0f1c9aaa795_isr_947244_39.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 751e99801dcc2246d8e6b14436ae12fc parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5178d66e-58fb-4738-ab0d-0e9d603fc26d_isr_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5178d66e-58fb-4738-ab0d-0e9d603fc26d_isr_947144_50.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 3d9b43f71c7b75c2f6936a9b7de980af parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c93a326c-b3d4-49db-9f5c-e24e7a964d3b_isr_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c93a326c-b3d4-49db-9f5c-e24e7a964d3b_isr_947246_5.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 0db98b0065fc8fb07b955e1ae8c55239 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b0e99148-1279-46c4-8ecd-bfed5d41edfb_isr_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b0e99148-1279-46c4-8ecd-bfed5d41edfb_isr_947244_5.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 249266b471bb429a1842d49f2ec33e7d parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f7995e2-e776-4046-a175-9f743d58d1fb_isr_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f7995e2-e776-4046-a175-9f743d58d1fb_isr_947144_28.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash a83086d6059c2cadda9859d394af33c2 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d0513c1f-ea9d-48a7-9176-33afcc27beea_isr_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d0513c1f-ea9d-48a7-9176-33afcc27beea_isr_947144_43.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash fff3e0feca8ea81a19e27cd3eeffce15 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2a3c7934-bf77-434b-b495-8e55f91ec1fa_isr_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2a3c7934-bf77-434b-b495-8e55f91ec1fa_isr_947144_52.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 8f52419d830fb8bc647aeb43af18f852 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6430f31-0561-445f-be2b-a7cc69e0e933_isr_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6430f31-0561-445f-be2b-a7cc69e0e933_isr_947244_13.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 1272d3bf236b40f09a518c41e9108bb9 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/c480c453-8489-41ed-bb76-e9e7a8c86cd3_isr_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/c480c453-8489-41ed-bb76-e9e7a8c86cd3_isr_947144_55.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash a4033ac9fab88cf9ab26dd0a90f7d207 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9cbb403f-4e0b-42ff-8863-046600e5a45b_isr_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9cbb403f-4e0b-42ff-8863-046600e5a45b_isr_947144_14.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash d386db544230ad93f1abd502df248d82 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/81f855f0-625f-4a0f-b351-ba9fc82103e2_isr_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/81f855f0-625f-4a0f-b351-ba9fc82103e2_isr_947140_58.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 3c65efea861b72c77dca139687db0bb7 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9e97e117-8b9d-475d-9167-7051427b6173_isr_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9e97e117-8b9d-475d-9167-7051427b6173_isr_947248_57.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 1074d23e47d445267829f8bc5bd11a73 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/8b538745-9fea-4ba5-932c-4c896748af39_isr_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/8b538745-9fea-4ba5-932c-4c896748af39_isr_947246_24.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash fe0180c0c27e8a801b993c78a052e35c parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f996bb0b-e3dd-40e8-a185-1765148e0d0a_isr_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f996bb0b-e3dd-40e8-a185-1765148e0d0a_isr_947142_36.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 2888c3178f37c1ad72ffff6d0b301a9b parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/98b4ba79-9a1a-41e6-a0a0-2648e217bd32_isr_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/98b4ba79-9a1a-41e6-a0a0-2648e217bd32_isr_947246_59.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 5949ea0ff33a9e2d584705416cbe9bcb parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5a8bbb9c-da21-4ed8-ae5f-072a81fb35b6_isr_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5a8bbb9c-da21-4ed8-ae5f-072a81fb35b6_isr_947248_32.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 725ccb04ff787da4074760982aa0b357 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/291cc40b-b95c-4caa-9f41-3ea3bf7cc734_isr_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/291cc40b-b95c-4caa-9f41-3ea3bf7cc734_isr_947244_19.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash d7356c3ab4957930aa7edf957cd6cb53 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/ec277bd8-bde7-4777-8b26-6dcc2907d8a4_isr_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/ec277bd8-bde7-4777-8b26-6dcc2907d8a4_isr_947140_22.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 2b3b357873b10f09f6d7f92f2b0fec83 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/dd28e67e-a2dd-4856-b7a5-fb2e1eec81b8_isr_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/dd28e67e-a2dd-4856-b7a5-fb2e1eec81b8_isr_947144_61.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash c69ec2f60ffec5fcef1185296cc480c0 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/bddd4389-1e35-4fb3-8f4e-262b1691aef1_isr_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/bddd4389-1e35-4fb3-8f4e-262b1691aef1_isr_947248_33.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 25e6e8c637959f0c37cb5b2c9911b297 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f9f7925a-1676-410c-83c1-ab56f85d4c38_isr_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f9f7925a-1676-410c-83c1-ab56f85d4c38_isr_947140_29.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 8532686f0d44faa8786c182b5789ddbf parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17031551 proc_lsst.multi INFO: [multi] job_id 17031551 2024-03-14 05:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031551 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031551 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17031551 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-14 05:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/03a295b5-a469-4647-8ec9-e2443c244a9e_isr_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/03a295b5-a469-4647-8ec9-e2443c244a9e_isr_947248_16.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 1a597226f9b66bc15a2b8acc8d2dccba parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c69c67c8-910d-4dc0-b338-f1a98f8626a8_isr_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c69c67c8-910d-4dc0-b338-f1a98f8626a8_isr_947140_38.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash d77eeaa53372dbb5dd39a188d7d8f329 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8e9826b5-f085-4392-86c6-65879739b837_isr_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8e9826b5-f085-4392-86c6-65879739b837_isr_947244_31.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash fada3335da369abc8b234a16c7ad056e parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a38a6401-41fc-4c10-a06c-6929a9f0a453_isr_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a38a6401-41fc-4c10-a06c-6929a9f0a453_isr_947244_4.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash e333e541fe723a80cf2162774cf8dd69 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/baf044f8-20ef-4386-abf1-a6afc6d8c89f_isr_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/baf044f8-20ef-4386-abf1-a6afc6d8c89f_isr_947244_62.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 7241ce7e24096a6e42b0b01191bb8a79 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5f41e7da-d0e5-40ff-b5e9-2fcc9b8f84ff_isr_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5f41e7da-d0e5-40ff-b5e9-2fcc9b8f84ff_isr_947140_59.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash bd2b2e47628bb44b2e10aff293a555eb parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6af28d3-bdc9-4033-88da-36cab465c348_isr_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6af28d3-bdc9-4033-88da-36cab465c348_isr_947244_46.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 4abd5b19c58969ca377d021975d8b300 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8999eedd-6715-49b9-9beb-52a876d5472d_isr_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8999eedd-6715-49b9-9beb-52a876d5472d_isr_947144_58.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 2e7a038a2bb7a0af103922ddf77d79c8 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3413af2f-12a1-46b9-ab30-064437519f44_isr_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3413af2f-12a1-46b9-ab30-064437519f44_isr_947140_32.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash e19f614f6ed890e7c63e2fbe05d12b38 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68798a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a6b093ef-ff6b-4382-a5eb-36e1d3ab08e2_isr_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a6b093ef-ff6b-4382-a5eb-36e1d3ab08e2_isr_947244_59.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash bc8588acddebffdaba42c5b4a0119470 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/de354ad1-9735-43d1-bc88-f8a0ae2e565d_isr_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/de354ad1-9735-43d1-bc88-f8a0ae2e565d_isr_947144_37.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash ece016bcb5a27d879a5fa737411e2a0a parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5c674f8e-b9f1-4018-a2e4-346c5dde4c51_isr_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5c674f8e-b9f1-4018-a2e4-346c5dde4c51_isr_947244_32.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash f877229300a5b95d64bb29cbaa26f4f0 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/eae7ac19-a8d7-4113-8a40-65534e3e5c58_isr_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/eae7ac19-a8d7-4113-8a40-65534e3e5c58_isr_947142_25.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 7b10389c36933fcedcab07e582b4030b parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4f0d9898-70c0-49b9-bc58-4717dc01ff4f_isr_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4f0d9898-70c0-49b9-bc58-4717dc01ff4f_isr_947142_27.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 22c2147aaf14cc5c8de928d1dcb04433 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/6d928e6a-d79f-4c35-83cb-9d4911cb5b73_isr_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/6d928e6a-d79f-4c35-83cb-9d4911cb5b73_isr_947142_49.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 967d67a8ff9c3a1f049a576ec1a518a1 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/040ad73d-1da1-4a31-b29a-9fc0cbc83bba_isr_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/040ad73d-1da1-4a31-b29a-9fc0cbc83bba_isr_947246_29.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 9adaad61bc573ab2826f69d242c6122b parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/18c2432c-f862-4f9a-baf1-c3d360ba086f_isr_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/18c2432c-f862-4f9a-baf1-c3d360ba086f_isr_947140_42.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 9d0f611c578d6cbf272bd04a577cbf52 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3523b5f6-9d08-42fa-a463-602de59a2571_isr_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3523b5f6-9d08-42fa-a463-602de59a2571_isr_947244_42.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 27d5d8364a4e4f284e7e5d3bd66b1b92 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/87231100-8efc-4220-9d71-4ea26e433430_isr_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/87231100-8efc-4220-9d71-4ea26e433430_isr_947246_25.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 4e0b2394ba4930f552a02fedab75d167 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fb001287-5fa3-4bb3-a1fc-cbdd2536877c_isr_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fb001287-5fa3-4bb3-a1fc-cbdd2536877c_isr_947248_29.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 2f2944b69034b3e4335ded59476407f1 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68782c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/68fbb37f-0792-4074-a831-fa99e86b19a0_isr_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/68fbb37f-0792-4074-a831-fa99e86b19a0_isr_947248_52.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 082d6aebb1cf874e66035ababa09b800 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8fc99916-0b64-4915-bb8c-f155c85e8c99_isr_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8fc99916-0b64-4915-bb8c-f155c85e8c99_isr_947142_22.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 80b52a3d468c8497b980da275f98a5ee parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/17074363-505a-4278-b903-f64cb87dd1a2_isr_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/17074363-505a-4278-b903-f64cb87dd1a2_isr_947140_21.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 93e9725324a93da0ed1a5737e60584e6 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4b9dc4c7-ca60-40a9-83e5-b0915f1ab013_isr_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4b9dc4c7-ca60-40a9-83e5-b0915f1ab013_isr_947142_20.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 4a689d11e4cfc601aadc5d4bd6a7c0bd parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4aed5285-f529-4e2d-a417-783181998bf6_isr_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4aed5285-f529-4e2d-a417-783181998bf6_isr_947244_55.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash a7bf1b10d6b488cbafce8e1677284faf parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8a388c9e-e7b2-4add-8763-df1086513668_isr_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8a388c9e-e7b2-4add-8763-df1086513668_isr_947144_3.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 0d157e74cb7ac953d9264c56e8251431 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/11659fff-092e-431b-a41c-86ad94c6cc97_isr_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/11659fff-092e-431b-a41c-86ad94c6cc97_isr_947244_36.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 59a50d5c2dc1bcd5c3877c855c717be6 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/50b96b63-644a-4f6c-ad8e-daf408b9c201_isr_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/50b96b63-644a-4f6c-ad8e-daf408b9c201_isr_947140_46.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash a09794392a1038b74a5ce712df42ea57 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3deeabfd-29be-4b97-aefd-5fd1abe258b5_isr_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3deeabfd-29be-4b97-aefd-5fd1abe258b5_isr_947246_17.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 1af553b133ea3fe979b664a0c95d89e4 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/58382c53-700d-4462-b34d-7bf9ed91c25a_isr_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/58382c53-700d-4462-b34d-7bf9ed91c25a_isr_947244_30.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 3b84288950832b61903fffb3c9291817 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/e127fcdf-ac85-43cf-a1bf-4eed55bdfed5_isr_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/e127fcdf-ac85-43cf-a1bf-4eed55bdfed5_isr_947248_35.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 72867c3822856112d25076dd5a446c00 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/aedb9997-15eb-4a46-95aa-1b2c0bc08330_isr_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/aedb9997-15eb-4a46-95aa-1b2c0bc08330_isr_947140_11.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash abfb88b39ee6182968b57a3c84bf972f parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3a8d595a-f6bc-4b6e-8d5c-4fb0bc33420e_isr_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3a8d595a-f6bc-4b6e-8d5c-4fb0bc33420e_isr_947246_7.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash e121eb26954e3a3045646cea45b60e16 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b8bc19f0-8200-443e-90a5-be059f18cb76_isr_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b8bc19f0-8200-443e-90a5-be059f18cb76_isr_947140_34.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash f84752e95628c090a0148762bd7df375 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17031552 proc_lsst.multi INFO: [multi] job_id 17031552 2024-03-14 05:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031552 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031552 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17031552 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-14 05:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/98b15f33-ea70-446d-9ece-e1fff1b2ba0a_isr_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/98b15f33-ea70-446d-9ece-e1fff1b2ba0a_isr_947244_38.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash acb05ca39f687e981f118dbef6804782 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/3a379e05-9764-41ed-a8d4-7ea65648de27_isr_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/3a379e05-9764-41ed-a8d4-7ea65648de27_isr_947144_1.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 6d11f7eb82e82fdabfc76ef805ffda72 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/825af127-f636-406e-9b67-73e8cbb9aa19_isr_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/825af127-f636-406e-9b67-73e8cbb9aa19_isr_947140_61.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 75e970a62dde9021d3ae41a69cf20990 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/31f6fa17-00b4-421d-a291-cef089eaecbe_isr_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/31f6fa17-00b4-421d-a291-cef089eaecbe_isr_947142_26.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash a588b92bd096e3913c2ca8c6702b282c parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e89f7c41-79cb-4e67-83f9-739df5e05a50_isr_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e89f7c41-79cb-4e67-83f9-739df5e05a50_isr_947142_11.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 3c0143f443f73c09b15e5d746f9152f3 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/216ed1c8-ce38-4cdf-994b-301e4328cec3_isr_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/216ed1c8-ce38-4cdf-994b-301e4328cec3_isr_947248_53.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 546796494c5469468d406a73def2a635 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ae7b31db-6d01-4d8a-8c8d-8c4d4fdb015c_isr_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ae7b31db-6d01-4d8a-8c8d-8c4d4fdb015c_isr_947142_18.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 54700ba47a6aa89d05fe047703a988aa parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f6c970c9-682a-4d16-9c7b-22923d213cec_isr_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f6c970c9-682a-4d16-9c7b-22923d213cec_isr_947144_57.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 777d8874598f3f0af5de0b9ed8f53c86 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3a59f128-32a2-42a6-989f-2c09257c809f_isr_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3a59f128-32a2-42a6-989f-2c09257c809f_isr_947248_36.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 6eb8c163a3bc0c4ae5fd5803d5c22a86 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/49c1d9fe-210c-478a-8a54-83c10463c986_isr_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/49c1d9fe-210c-478a-8a54-83c10463c986_isr_947248_13.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash f957f275772332001a3599b053ae9aff parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f7e1bdc4-5082-4395-87ab-3490ea31a102_isr_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f7e1bdc4-5082-4395-87ab-3490ea31a102_isr_947244_3.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash c1f20d73230298a5c949aeae0048bdca parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cb99c040-07d2-45a1-9582-ec5da6f488b5_isr_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cb99c040-07d2-45a1-9582-ec5da6f488b5_isr_947248_54.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 31140d3b03eb2e796a140a62d33409bc parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b09b5b14-4fdc-4126-abc6-29902f965c94_isr_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b09b5b14-4fdc-4126-abc6-29902f965c94_isr_947140_14.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash f91a3a16785272eda7aecacc905588e0 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/86e37b70-db96-449d-8b37-f3e1461050e6_isr_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/86e37b70-db96-449d-8b37-f3e1461050e6_isr_947248_20.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 80e245f357da91b9652e9d78b96ac90f parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/6ad6819d-8d66-43af-8e80-73f93259c85f_isr_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/6ad6819d-8d66-43af-8e80-73f93259c85f_isr_947144_30.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash b6d453d295fad4d0bc85fe31c2277505 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/015d5a00-9360-4b3f-a129-98c69c0103f8_isr_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/015d5a00-9360-4b3f-a129-98c69c0103f8_isr_947244_16.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 65ff2deb6515d1abdd9b6f64f47da75e parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2fe457d3-0436-4317-873b-0e254f577531_isr_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2fe457d3-0436-4317-873b-0e254f577531_isr_947142_57.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 3bc955373e967021a7afbe85825dabbe parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0ac8c0c6-7e5d-463c-b1bd-068b1a62dc14_isr_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0ac8c0c6-7e5d-463c-b1bd-068b1a62dc14_isr_947144_56.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 4a64bc3f8b84ed5f629fe9af30a15d1d parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d9664ae4-1f1f-4555-adf0-ebb03d286323_isr_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d9664ae4-1f1f-4555-adf0-ebb03d286323_isr_947248_50.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash f0cc8693dd21b19f7c37d053599c8c98 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5a3717a6-7b6d-4210-a234-2bec1f8a156e_isr_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5a3717a6-7b6d-4210-a234-2bec1f8a156e_isr_947244_54.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash e70473224db55389c523753613982afa parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/107b7c20-a87c-45a4-99c2-58517d5108e9_isr_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/107b7c20-a87c-45a4-99c2-58517d5108e9_isr_947144_36.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash a80048d9090b25d2801237da2b21bcd5 parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d00a7871-9807-43b5-8f7f-63a57bdec031_isr_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d00a7871-9807-43b5-8f7f-63a57bdec031_isr_947248_9.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash da2131844d9447a3d3f01ad3d609bf4d parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/432360be-f90f-46f8-8dbb-d49c5cd0e63c_isr_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/432360be-f90f-46f8-8dbb-d49c5cd0e63c_isr_947248_41.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash a9910d60dc22f208bb7da6bdd5207b63 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/414b6a08-a3e3-4385-ab97-b96d7502b1d8_isr_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/414b6a08-a3e3-4385-ab97-b96d7502b1d8_isr_947248_11.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash ed68398ece23fad0069e8cbb0deee7ea parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/31f35337-b9ed-4a4c-87a0-68b08b227ba6_isr_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/31f35337-b9ed-4a4c-87a0-68b08b227ba6_isr_947244_60.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 5d6c317216fdba2b05b1a4bc02f34058 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/19a51520-4b31-423f-a7be-a5801631e41e_isr_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/19a51520-4b31-423f-a7be-a5801631e41e_isr_947244_18.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash d0b6657f00b6dd089ebaa960121599b2 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/12493d1e-22c8-4501-9b74-37f9212f1e19_isr_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/12493d1e-22c8-4501-9b74-37f9212f1e19_isr_947144_45.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 5acf827ebe4df31b2e3682053b6dda36 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d543f47c-28be-4e8a-a870-32eae17811a2_isr_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d543f47c-28be-4e8a-a870-32eae17811a2_isr_947140_24.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 7b04dd6a834331fdaf38cef588586a24 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/41b62492-5a45-4612-be97-28ab489c9587_isr_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/41b62492-5a45-4612-be97-28ab489c9587_isr_947244_20.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 219c4fecb5caf679651eac6986ea6727 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/19e37b87-bc99-4fef-87c3-f3984826f6b0_isr_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/19e37b87-bc99-4fef-87c3-f3984826f6b0_isr_947142_13.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash e4bcb2bafaa0dc6af07d87e87549c9f4 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/76819225-79be-453d-b6cb-232fabafb428_isr_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/76819225-79be-453d-b6cb-232fabafb428_isr_947244_50.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash a2513e16dbeb54c2663b3d5af9cf0189 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f0241327-9c15-4637-a75e-cc31f86d3c40_isr_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f0241327-9c15-4637-a75e-cc31f86d3c40_isr_947144_10.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash a08d122851e62203078d00491ed8d5cd parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/82702a4f-6413-436d-aa99-034f135f1abc_isr_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/82702a4f-6413-436d-aa99-034f135f1abc_isr_947142_42.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 7c96f3e3d7a5ff4d79efe1af18ae5bc1 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b969e660-cb72-49d5-b4d2-e9a99ab0b774_isr_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b969e660-cb72-49d5-b4d2-e9a99ab0b774_isr_947142_54.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 6641748591c9e1672d859d0e17423353 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e4493787-4b40-4a5a-841a-278baa6eb347_isr_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e4493787-4b40-4a5a-841a-278baa6eb347_isr_947142_43.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 3b7ce7c1de5fa66169f6a1f03ae5e1f8 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3db45e4a-83db-4d52-9ec4-8a1f10ffe3b6_isr_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3db45e4a-83db-4d52-9ec4-8a1f10ffe3b6_isr_947140_17.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash c8ba01e51065950148e8debb2d7162c4 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17031553 proc_lsst.multi INFO: [multi] job_id 17031553 2024-03-14 05:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031553 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031553 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17031553 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-14 05:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c5e8e0db-7cc4-40c3-b771-244bfc7ce810_isr_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c5e8e0db-7cc4-40c3-b771-244bfc7ce810_isr_947246_57.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 727669731e586986e3da4a3afcde6ede parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a285015f-3a50-43a8-a82f-4eb86819d0fa_isr_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a285015f-3a50-43a8-a82f-4eb86819d0fa_isr_947142_39.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash de8e78aa67ba51b97363eb98687db208 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/cf61768d-0899-48c8-b8f7-ebe0940077f0_isr_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/cf61768d-0899-48c8-b8f7-ebe0940077f0_isr_947142_17.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 0b57848ace7e8d05cb15c667fe33b544 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2c8745ef-0e0f-48dd-a1cf-4a6651ed277d_isr_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2c8745ef-0e0f-48dd-a1cf-4a6651ed277d_isr_947144_9.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 749d43316d78ccb59052b25b223e4888 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2911c989-713e-427f-bf71-6f4c220eea71_isr_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2911c989-713e-427f-bf71-6f4c220eea71_isr_947246_18.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 4aca6265ccbe032b7884ee14c52e3329 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b6a14c86-7052-44ba-864c-7b9fad19f07a_isr_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b6a14c86-7052-44ba-864c-7b9fad19f07a_isr_947246_38.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash c6e1c6bba2d415214a57887c1a9b0aeb parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/24d0cfc5-8a91-442f-ba64-4167ddccd516_isr_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/24d0cfc5-8a91-442f-ba64-4167ddccd516_isr_947246_32.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash baac75837c7f21e412bc560be645291d parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/dbe88134-dc97-4088-97fc-927cc9b9422d_isr_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/dbe88134-dc97-4088-97fc-927cc9b9422d_isr_947248_6.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 814155bc49f1f1c883dcdf63df297324 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e41a827f-98b5-4fa1-94ae-d076b9f1d167_isr_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e41a827f-98b5-4fa1-94ae-d076b9f1d167_isr_947144_13.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 22c171482327fbc78b64b98dab928d39 parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5ca77c97-9608-4b25-a308-c5103b7e9d6f_isr_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5ca77c97-9608-4b25-a308-c5103b7e9d6f_isr_947142_51.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 6d8962ddc815f6265faeec4b113e986d parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ac82363b-913d-432b-b437-e4287a56bd2c_isr_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ac82363b-913d-432b-b437-e4287a56bd2c_isr_947142_48.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash e284bb647ed9e736986139cf55668349 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b3872193-4bd4-4ec3-adf3-3427e826506c_isr_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b3872193-4bd4-4ec3-adf3-3427e826506c_isr_947140_55.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash b4d294cf09156d77f3f3dcc57ab8c4cd parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/296a9cc9-d726-4517-8c65-5c4b419b2df3_isr_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/296a9cc9-d726-4517-8c65-5c4b419b2df3_isr_947246_42.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash b5f81e16e7eb191f7d89e0f7f499b3ab parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af5ebc1f-03d1-4d26-b3e1-44ac3d82cd5b_isr_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af5ebc1f-03d1-4d26-b3e1-44ac3d82cd5b_isr_947142_50.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash bde2785b4c71ae119393cb2d3706eca7 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b48e5ba3-cb44-4de2-8c11-6a7cba6aaf7b_isr_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b48e5ba3-cb44-4de2-8c11-6a7cba6aaf7b_isr_947246_4.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash a0cfa233d6b4c37d69f5aa12c10d7cac parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1f5b8af7-5759-4679-b85d-26a7160a82e9_isr_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1f5b8af7-5759-4679-b85d-26a7160a82e9_isr_947142_4.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 3d5d6e0f1af6bb21d24385eaceb702c3 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8314b878-1242-49f9-8a2e-fdc5f3c16ed6_isr_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8314b878-1242-49f9-8a2e-fdc5f3c16ed6_isr_947248_42.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 82227befa00e3a418225cc36831ddcae parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f198ccfa-0142-4cee-819a-b2b5d1803719_isr_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f198ccfa-0142-4cee-819a-b2b5d1803719_isr_947244_11.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 83a1433112d1d4c373afd6276496d885 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/835dceee-4308-4813-be50-39ba3d4f5e10_isr_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/835dceee-4308-4813-be50-39ba3d4f5e10_isr_947144_2.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 0cc8dbf7696b7c192a031dbdeb5c999b parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8c5ce78e-efdf-4677-9a62-3613d83912ef_isr_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8c5ce78e-efdf-4677-9a62-3613d83912ef_isr_947142_16.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash a7b6eb9b8e84b702b5cf3a30ec9d47af parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c589dcef-cf3a-43f4-b05c-15dd97c056f8_isr_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c589dcef-cf3a-43f4-b05c-15dd97c056f8_isr_947140_56.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash b24fa3ba9028760cb775d3deefc36a01 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4af2aa3a-fea4-424b-81e7-a8eff2d19c65_isr_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4af2aa3a-fea4-424b-81e7-a8eff2d19c65_isr_947248_34.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 8f386766aced367d89f1800bab88c80a parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/13d52003-6524-4a43-9cc5-df8ae91884b4_isr_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/13d52003-6524-4a43-9cc5-df8ae91884b4_isr_947248_51.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 98e4b24d714969f203163326d28fc49d parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cde92c-7aa2-4030-add1-56acddbda51f_isr_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cde92c-7aa2-4030-add1-56acddbda51f_isr_947244_12.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 7ae6b0a260bae436e47e1f4fdb6c5b90 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c03115f2-027c-4fba-85da-e2c563fa08df_isr_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c03115f2-027c-4fba-85da-e2c563fa08df_isr_947246_31.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 426d967fa8e4af4e3f74c18489b00849 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/913454f7-a3c2-47e8-a7f2-bc3312677f7c_isr_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/913454f7-a3c2-47e8-a7f2-bc3312677f7c_isr_947246_20.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash d407640a36dcfba619eac8ffa1abba93 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a8c30454-3dcd-4e7c-895d-bbb6351fdad7_isr_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a8c30454-3dcd-4e7c-895d-bbb6351fdad7_isr_947140_15.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 4dabf610ade9ebc455dd43746d4759c8 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4870ad0b-8565-4aa5-ab74-567441e799b0_isr_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4870ad0b-8565-4aa5-ab74-567441e799b0_isr_947244_61.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 718ff1c391de941f74b2a536c2afbdb9 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2f7f8969-3b1f-425b-bdd1-48486ef26c07_isr_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2f7f8969-3b1f-425b-bdd1-48486ef26c07_isr_947140_26.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash f84865613e457d4722c716ae0ee87ac0 parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/10c97cad-771b-4592-ba99-f309670bf25d_isr_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/10c97cad-771b-4592-ba99-f309670bf25d_isr_947248_46.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 3639e7aaf84745c4b0e6b738118972d7 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6ec75fa0-a1dc-4eb4-a819-d7b21ba3d141_isr_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6ec75fa0-a1dc-4eb4-a819-d7b21ba3d141_isr_947244_27.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash e55cc8b612a5d0bdfbc4d8340e5c175d parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/64bfd8ce-e1a8-41d8-9b51-239b63311823_isr_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/64bfd8ce-e1a8-41d8-9b51-239b63311823_isr_947140_48.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash fd1e1d78254c0cc5385f83312e2b6d1e parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67accc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3fbacf7-d3b1-4537-8936-9ad374528d1c_isr_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3fbacf7-d3b1-4537-8936-9ad374528d1c_isr_947244_58.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 8f38cc47ceb31133cd52c3ac5a890e9b parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67acfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1bd158da-3381-4183-b205-4233de44aaee_isr_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1bd158da-3381-4183-b205-4233de44aaee_isr_947246_61.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash da3256363a968e15516363970da84876 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/620f54bf-5e3f-45b1-ba66-1c20f38c8172_isr_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/620f54bf-5e3f-45b1-ba66-1c20f38c8172_isr_947140_35.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 6df9f9aae8d78acfb7f780ed16fd44e9 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/da6b4f0f-dd4c-4498-8ae0-714ee1900a2c_isr_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/da6b4f0f-dd4c-4498-8ae0-714ee1900a2c_isr_947246_12.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash d19eefae961224b300fee5892888fa7b parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63c6422f-d002-41ff-96b7-9c7934124716_isr_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63c6422f-d002-41ff-96b7-9c7934124716_isr_947244_41.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 881bdcc41c86beae669405f902da204d parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45994b8-800a-4eb7-b938-8f4e5001e140_isr_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45994b8-800a-4eb7-b938-8f4e5001e140_isr_947244_15.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash a2d25d3b1bb1685dfbf28cd4a4a0cd9e parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67adf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/94aab505-0720-4cf5-9bbf-d903fddbcf69_isr_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/94aab505-0720-4cf5-9bbf-d903fddbcf69_isr_947144_48.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 4cede873dcc037dabdf375bbb2a148e1 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47b4174d-cd11-4840-b9ee-454106a85536_isr_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47b4174d-cd11-4840-b9ee-454106a85536_isr_947248_62.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 4126ca1559a15a87149c3fe836f440fe parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/84f1bf39-b1f0-4706-bab2-5f69c8038f02_isr_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/84f1bf39-b1f0-4706-bab2-5f69c8038f02_isr_947246_28.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash f1375a72229f9cb1bac3113f5f08916a parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/905c0732-6424-4a0c-9a8a-0f36ea4df6d2_isr_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/905c0732-6424-4a0c-9a8a-0f36ea4df6d2_isr_947140_40.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash b0e4ecf727154a5386fe4f32e975146d parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f5cbd07-3c53-4a0e-bd5a-f6d328500ecd_isr_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f5cbd07-3c53-4a0e-bd5a-f6d328500ecd_isr_947140_44.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 8d5b221de960af4fd052e16b4b219072 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8a5188c5-6ca6-45c4-8b63-cc98f764c11b_isr_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8a5188c5-6ca6-45c4-8b63-cc98f764c11b_isr_947244_43.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash e38351386f2bc1cf333208086e79f5b5 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/84125a66-090c-4550-b533-818805b4dcc2_isr_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/84125a66-090c-4550-b533-818805b4dcc2_isr_947244_40.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 7fe00bd2c00ca2210d036dc591f26482 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f4a207d1-eb59-4d78-9bbe-d9a6d0539b6b_isr_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f4a207d1-eb59-4d78-9bbe-d9a6d0539b6b_isr_947140_16.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash e8545b29608a3d2ece0be3ccd8a4986a parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d0680c78-2af3-4d50-9493-0774cafeecb7_isr_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d0680c78-2af3-4d50-9493-0774cafeecb7_isr_947140_9.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash d8e9f31e1896a8a0f86aedd4fb726350 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67afba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/46b75715-6cbe-460d-9631-6cb9430e4ef2_isr_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/46b75715-6cbe-460d-9631-6cb9430e4ef2_isr_947144_26.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash a7b89c7a6309bf5dcd3f0aaae7867b9e parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/427390eb-fe14-4492-9f5c-954cbb2602cc_isr_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/427390eb-fe14-4492-9f5c-954cbb2602cc_isr_947144_15.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 27f7cbc01a6169c45cf40f6cbcdcab7b parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d29f70f5-de9c-4d25-ac41-948bfecede10_isr_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d29f70f5-de9c-4d25-ac41-948bfecede10_isr_947246_37.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash f7bb9dc31f008e51f01d9083fd9f1e64 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/99bc3611-cad1-4d02-b1db-256d59e89738_isr_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/99bc3611-cad1-4d02-b1db-256d59e89738_isr_947144_20.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 6958d5eb68068b7d312f77cd26af1811 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17031554 proc_lsst.multi INFO: [multi] job_id 17031554 2024-03-14 05:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031554 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031554 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17031554 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-14 05:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/57b9bae6-526e-40a0-bd8e-10122370da5a_isr_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/57b9bae6-526e-40a0-bd8e-10122370da5a_isr_947142_38.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 1b042258687b06be9c347bfc6a4d04f6 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3167c557-e9fa-4ab9-a144-1c744dfd59a2_isr_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3167c557-e9fa-4ab9-a144-1c744dfd59a2_isr_947142_32.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 098d809994c501a6b5de07cf9f61b613 parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/913b01e4-340e-4943-9d1e-069cbb94e53b_isr_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/913b01e4-340e-4943-9d1e-069cbb94e53b_isr_947142_29.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 29850869d47375a155c1f7defff9db0d parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c696e03e-b245-4ac1-a8f9-deac03348146_isr_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c696e03e-b245-4ac1-a8f9-deac03348146_isr_947246_1.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash e3a5e748744aba019ef121e8fd8ea7ca parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c7025827-aea3-4a4d-bcdc-4e4928fc3cf1_isr_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c7025827-aea3-4a4d-bcdc-4e4928fc3cf1_isr_947142_21.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 04121112e46194a5fd3a72f315ba85cc parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cce60aa9-f827-47c6-a290-efa574d8612a_isr_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cce60aa9-f827-47c6-a290-efa574d8612a_isr_947248_47.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash b6772a4a91d011d7e2da3ee7f86d8977 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/78a7c94e-9303-4bce-bf11-4b4b75170103_isr_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/78a7c94e-9303-4bce-bf11-4b4b75170103_isr_947144_47.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash bc6c16f55ae615bec28ea3d6ac4932a3 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b6017c66-c90e-45e9-b10e-c7a45ab7201d_isr_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b6017c66-c90e-45e9-b10e-c7a45ab7201d_isr_947142_46.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash d8784016a1d36d6da19b5eb1baaefeec parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7e84777d-7664-45c6-b2ed-daf6e18be943_isr_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7e84777d-7664-45c6-b2ed-daf6e18be943_isr_947142_31.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 1e4c68c9242db8ec7ac449a8382f8723 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/610a2081-e60e-409f-acbd-ced7ac3e0a89_isr_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/610a2081-e60e-409f-acbd-ced7ac3e0a89_isr_947142_30.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash f0b2ec95f609c09607e5137be3e04f85 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/db42beb9-ae33-4596-878f-b16083d7b2b7_isr_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/db42beb9-ae33-4596-878f-b16083d7b2b7_isr_947140_47.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 0a58b393db5edb66a2984c7e6ee62588 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/db86b8a3-94c4-4cb7-a8c0-04fa9e7ba9b2_isr_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/db86b8a3-94c4-4cb7-a8c0-04fa9e7ba9b2_isr_947144_53.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 25f12927a2898aadcde17ca131e8ee2a parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/1e5c22fd-eb3a-468d-9598-daf6634fbb37_isr_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/1e5c22fd-eb3a-468d-9598-daf6634fbb37_isr_947144_22.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash fb340cc292b2018f00e1666bb33bb0c7 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/66d2df00-c846-4b54-a324-33fb0b2e74f4_isr_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/66d2df00-c846-4b54-a324-33fb0b2e74f4_isr_947144_21.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 1be57bc287d6361c0a1d4c016354093e parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67faa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f2e218e4-78e2-4e46-9416-45fea67f0544_isr_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f2e218e4-78e2-4e46-9416-45fea67f0544_isr_947246_16.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 3e2379fe0c9fa15cc62e2b727144f445 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a5ec563d-821c-4bf7-9472-193076c61fd1_isr_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a5ec563d-821c-4bf7-9472-193076c61fd1_isr_947140_52.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash b1c011d570423e4f0f8d36499bcfcc03 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f135a2aa-c7c7-4e7d-ba96-2a8e12b8a38d_isr_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f135a2aa-c7c7-4e7d-ba96-2a8e12b8a38d_isr_947244_8.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash a8cfffc3f04de8ee3750e3453724d42e parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/48c79386-942d-42ce-9789-6af3a292e133_isr_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/48c79386-942d-42ce-9789-6af3a292e133_isr_947246_9.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash a2d6d2afc5ee7a67e6e93bb69c008371 parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3ad558dd-c907-403b-9694-35182f26cdda_isr_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3ad558dd-c907-403b-9694-35182f26cdda_isr_947246_14.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 919a7aa10347d92e9c4631d570a2c7d6 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/661c6954-cfd0-41a3-89fb-6dec652eb296_isr_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/661c6954-cfd0-41a3-89fb-6dec652eb296_isr_947246_43.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 1b865e878a042766a0634eb0e17efbad parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/601098ca-6a6e-46ca-bb16-a530b21ab962_isr_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/601098ca-6a6e-46ca-bb16-a530b21ab962_isr_947248_39.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash b184311414e4733f77cd325d72804d65 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c52e4deb-4336-4b7f-be81-10c2010ce3a2_isr_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c52e4deb-4336-4b7f-be81-10c2010ce3a2_isr_947140_8.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash bc0d4f128780c052c48a61e574011148 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/30b7765d-3ed9-486d-ad52-3a8be984ec58_isr_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/30b7765d-3ed9-486d-ad52-3a8be984ec58_isr_947144_31.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash c4812fdb01a50fd0793fb5ce292a48cf parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/13215e26-02be-4c66-abb2-2f05c62e3c98_isr_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/13215e26-02be-4c66-abb2-2f05c62e3c98_isr_947140_2.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash cf23303f2e56cb9456f11168d8bdf587 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a645c62f-42b4-4d5b-988d-7f0eafd09ccf_isr_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a645c62f-42b4-4d5b-988d-7f0eafd09ccf_isr_947246_22.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 73a9e97644e9f6974430d3e226eac222 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/382227f5-ca6f-4989-a371-5c581be30b7c_isr_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/382227f5-ca6f-4989-a371-5c581be30b7c_isr_947144_7.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash deb44920fbc6101668d06d7448d78937 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7759db30-2bea-4d0f-9f93-d890e5340f3f_isr_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7759db30-2bea-4d0f-9f93-d890e5340f3f_isr_947244_37.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash da2975a5f0bdb9a74dd0e1bb06d2f5da parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/07110e11-eb46-41b5-9670-28ac6a9faa35_isr_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/07110e11-eb46-41b5-9670-28ac6a9faa35_isr_947140_45.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash a20a2db64e3f7f3ddb02a3bd0ec0a2d1 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6636020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4d43a621-5613-4af5-a413-d92de164a031_isr_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4d43a621-5613-4af5-a413-d92de164a031_isr_947244_53.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 60f02b6ca185aa414a03b3f37d9908f9 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6636340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/cbffd831-57d8-45e0-bfc7-54e885ac88bf_isr_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/cbffd831-57d8-45e0-bfc7-54e885ac88bf_isr_947246_49.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-14 05:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17031555 proc_lsst.multi INFO: [multi] job_id 17031555 2024-03-14 05:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031555 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031555 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17031555 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-14 05:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17031556 proc_lsst.multi INFO: [multi] job_id 17031556 2024-03-14 05:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031556 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031556 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17031556 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-14 05:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17031557 proc_lsst.multi INFO: [multi] job_id 17031557 2024-03-14 05:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031557 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031557 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17031557 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-14 05:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17031558 proc_lsst.multi INFO: [multi] job_id 17031558 2024-03-14 05:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031558 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031558 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17031558 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-14 05:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17031559 proc_lsst.multi INFO: [multi] job_id 17031559 2024-03-14 05:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031559 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031559 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17031559 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-14 05:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17031560 proc_lsst.multi INFO: [multi] job_id 17031560 2024-03-14 05:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031560 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031560 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17031560 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-14 05:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17031561 proc_lsst.multi INFO: [multi] job_id 17031561 2024-03-14 05:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031561 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031561 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17031561 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-14 05:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:17 proc_lsst.multi:170 [INFO] [multi] job_id 17031562 proc_lsst.multi INFO: [multi] job_id 17031562 2024-03-14 05:33:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031562 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031562 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17031562 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-14 05:33:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-14 05:33:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:17 proc_lsst.multi:170 [INFO] [multi] job_id 17031563 proc_lsst.multi INFO: [multi] job_id 17031563 2024-03-14 05:33:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031563 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031563 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17031563 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 17, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/16 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 17 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 76 more blocks parsl.executors.status_handling INFO: Scaling out by 76 blocks parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-14 05:33:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-14 05:33:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17031564 proc_lsst.multi INFO: [multi] job_id 17031564 2024-03-14 05:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031564 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031564 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17031564 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-14 05:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17031565 proc_lsst.multi INFO: [multi] job_id 17031565 2024-03-14 05:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031565 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031565 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17031565 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-14 05:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17031566 proc_lsst.multi INFO: [multi] job_id 17031566 2024-03-14 05:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031566 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031566 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17031566 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-14 05:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17031567 proc_lsst.multi INFO: [multi] job_id 17031567 2024-03-14 05:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031567 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031567 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17031567 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-14 05:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17031568 proc_lsst.multi INFO: [multi] job_id 17031568 2024-03-14 05:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 05:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031568 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031568 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17031568 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-14 05:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17031569 proc_lsst.multi INFO: [multi] job_id 17031569 2024-03-14 05:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 05:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031569 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031569 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17031569 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-14 05:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17031570 proc_lsst.multi INFO: [multi] job_id 17031570 2024-03-14 05:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 05:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031570 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031570 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17031570 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-14 05:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17031571 proc_lsst.multi INFO: [multi] job_id 17031571 2024-03-14 05:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 05:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031571 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031571 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17031571 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-14 05:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 05:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031572 proc_lsst.multi INFO: [multi] job_id 17031572 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031572 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031572 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17031572 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031573 proc_lsst.multi INFO: [multi] job_id 17031573 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031573 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031573 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17031573 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031574 proc_lsst.multi INFO: [multi] job_id 17031574 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031574 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031574 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17031574 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031575 proc_lsst.multi INFO: [multi] job_id 17031575 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031575 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031575 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17031575 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031576 proc_lsst.multi INFO: [multi] job_id 17031576 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031576 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031576 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17031576 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031577 proc_lsst.multi INFO: [multi] job_id 17031577 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17031577 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17031577 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17031577 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031578 proc_lsst.multi INFO: [multi] job_id 17031578 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17031578 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17031578 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17031578 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031579 proc_lsst.multi INFO: [multi] job_id 17031579 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17031579 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17031579 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17031579 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17031580 proc_lsst.multi INFO: [multi] job_id 17031580 2024-03-14 05:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17031580 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17031580 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17031580 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-14 05:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031581 proc_lsst.multi INFO: [multi] job_id 17031581 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17031581 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17031581 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17031581 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031582 proc_lsst.multi INFO: [multi] job_id 17031582 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17031582 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17031582 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17031582 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031583 proc_lsst.multi INFO: [multi] job_id 17031583 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031583 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031583 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17031583 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031584 proc_lsst.multi INFO: [multi] job_id 17031584 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031584 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031584 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17031584 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031585 proc_lsst.multi INFO: [multi] job_id 17031585 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031585 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031585 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17031585 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17031586 proc_lsst.multi INFO: [multi] job_id 17031586 2024-03-14 05:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031586 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031586 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17031586 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-14 05:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17031587 proc_lsst.multi INFO: [multi] job_id 17031587 2024-03-14 05:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031587 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031587 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17031587 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-14 05:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17031588 proc_lsst.multi INFO: [multi] job_id 17031588 2024-03-14 05:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031588 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031588 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17031588 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-14 05:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17031589 proc_lsst.multi INFO: [multi] job_id 17031589 2024-03-14 05:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031589 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031589 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17031589 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-14 05:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17031590 proc_lsst.multi INFO: [multi] job_id 17031590 2024-03-14 05:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031590 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031590 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17031590 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-14 05:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17031591 proc_lsst.multi INFO: [multi] job_id 17031591 2024-03-14 05:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031591 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031591 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17031591 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-14 05:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17031592 proc_lsst.multi INFO: [multi] job_id 17031592 2024-03-14 05:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031592 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031592 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17031592 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-14 05:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17031593 proc_lsst.multi INFO: [multi] job_id 17031593 2024-03-14 05:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031593 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031593 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17031593 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-14 05:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17031594 proc_lsst.multi INFO: [multi] job_id 17031594 2024-03-14 05:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031594 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031594 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17031594 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-14 05:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17031595 proc_lsst.multi INFO: [multi] job_id 17031595 2024-03-14 05:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031595 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031595 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17031595 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-14 05:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17031596 proc_lsst.multi INFO: [multi] job_id 17031596 2024-03-14 05:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031596 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031596 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17031596 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-14 05:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17031597 proc_lsst.multi INFO: [multi] job_id 17031597 2024-03-14 05:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031597 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031597 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17031597 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-14 05:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17031598 proc_lsst.multi INFO: [multi] job_id 17031598 2024-03-14 05:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031598 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031598 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17031598 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-14 05:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17031599 proc_lsst.multi INFO: [multi] job_id 17031599 2024-03-14 05:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031599 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031599 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17031599 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-14 05:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17031600 proc_lsst.multi INFO: [multi] job_id 17031600 2024-03-14 05:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031600 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031600 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17031600 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-14 05:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17031601 proc_lsst.multi INFO: [multi] job_id 17031601 2024-03-14 05:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031601 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031601 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17031601 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-14 05:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17031602 proc_lsst.multi INFO: [multi] job_id 17031602 2024-03-14 05:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17031602 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17031602 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17031602 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-14 05:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17031603 proc_lsst.multi INFO: [multi] job_id 17031603 2024-03-14 05:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031603 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031603 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17031603 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-14 05:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17031604 proc_lsst.multi INFO: [multi] job_id 17031604 2024-03-14 05:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031604 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031604 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17031604 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-14 05:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17031605 proc_lsst.multi INFO: [multi] job_id 17031605 2024-03-14 05:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031605 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031605 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17031605 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-14 05:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17031606 proc_lsst.multi INFO: [multi] job_id 17031606 2024-03-14 05:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031606 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031606 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17031606 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-14 05:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17031607 proc_lsst.multi INFO: [multi] job_id 17031607 2024-03-14 05:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031607 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031607 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17031607 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-14 05:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17031608 proc_lsst.multi INFO: [multi] job_id 17031608 2024-03-14 05:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031608 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031608 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17031608 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-14 05:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c03c6b37366855d36fee4563e54f2a49 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 47f42fd227ac23982ae5d19796977b49 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eb0400d7-c930-44c6-8406-26070f78af02_characterizeImage_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eb0400d7-c930-44c6-8406-26070f78af02_characterizeImage_947244_57.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63fb02a1-d5d2-4e79-ad06-7fbd2eb92c1a_isr_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63fb02a1-d5d2-4e79-ad06-7fbd2eb92c1a_isr_947244_57.stderr 2024-03-14 05:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17031609 proc_lsst.multi INFO: [multi] job_id 17031609 2024-03-14 05:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031609 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031609 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17031609 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-14 05:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17031610 proc_lsst.multi INFO: [multi] job_id 17031610 2024-03-14 05:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031610 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031610 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17031610 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-14 05:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 05:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b63dc54c4c537274b7e4b1944f62e82 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 85bc2faf2cf616edfff57c0e7042f1b4 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/821cf2a1-23f1-4e35-9d66-7919b5ee9490_characterizeImage_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/821cf2a1-23f1-4e35-9d66-7919b5ee9490_characterizeImage_947246_44.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a6edb0d7-8342-4686-9c01-09db2e63b3a2_isr_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a6edb0d7-8342-4686-9c01-09db2e63b3a2_isr_947246_44.stderr 2024-03-14 05:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17031611 proc_lsst.multi INFO: [multi] job_id 17031611 2024-03-14 05:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031611 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031611 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17031611 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-14 05:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17031612 proc_lsst.multi INFO: [multi] job_id 17031612 2024-03-14 05:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031612 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031612 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17031612 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-14 05:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17031613 proc_lsst.multi INFO: [multi] job_id 17031613 2024-03-14 05:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031613 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031613 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17031613 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-14 05:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17031614 proc_lsst.multi INFO: [multi] job_id 17031614 2024-03-14 05:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031614 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031614 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17031614 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-14 05:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 05:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17031615 proc_lsst.multi INFO: [multi] job_id 17031615 2024-03-14 05:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031615 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031615 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17031615 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-14 05:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17031616 proc_lsst.multi INFO: [multi] job_id 17031616 2024-03-14 05:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031616 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031616 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17031616 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-14 05:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17031617 proc_lsst.multi INFO: [multi] job_id 17031617 2024-03-14 05:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031617 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031617 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17031617 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-14 05:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 05:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17031618 proc_lsst.multi INFO: [multi] job_id 17031618 2024-03-14 05:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031618 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031618 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17031618 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-14 05:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17031619 proc_lsst.multi INFO: [multi] job_id 17031619 2024-03-14 05:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031619 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031619 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17031619 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-14 05:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17031620 proc_lsst.multi INFO: [multi] job_id 17031620 2024-03-14 05:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031620 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031620 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17031620 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-14 05:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 05:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17031621 proc_lsst.multi INFO: [multi] job_id 17031621 2024-03-14 05:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031621 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031621 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17031621 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-14 05:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17031622 proc_lsst.multi INFO: [multi] job_id 17031622 2024-03-14 05:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031622 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031622 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17031622 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-14 05:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17031623 proc_lsst.multi INFO: [multi] job_id 17031623 2024-03-14 05:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 05:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031623 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031623 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17031623 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-14 05:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17031624 proc_lsst.multi INFO: [multi] job_id 17031624 2024-03-14 05:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 05:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031624 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031624 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17031624 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-14 05:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17031625 proc_lsst.multi INFO: [multi] job_id 17031625 2024-03-14 05:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 05:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031625 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031625 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17031625 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-14 05:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 05:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031626 proc_lsst.multi INFO: [multi] job_id 17031626 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031626 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031626 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17031626 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031627 proc_lsst.multi INFO: [multi] job_id 17031627 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031627 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031627 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17031627 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031628 proc_lsst.multi INFO: [multi] job_id 17031628 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031628 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031628 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17031628 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031629 proc_lsst.multi INFO: [multi] job_id 17031629 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031629 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031629 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17031629 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031630 proc_lsst.multi INFO: [multi] job_id 17031630 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031630 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031630 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17031630 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031631 proc_lsst.multi INFO: [multi] job_id 17031631 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031631 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031631 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17031631 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17031632 proc_lsst.multi INFO: [multi] job_id 17031632 2024-03-14 05:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031632 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031632 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17031632 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-14 05:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17031633 proc_lsst.multi INFO: [multi] job_id 17031633 2024-03-14 05:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-14 05:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031633 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031633 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17031633 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-14 05:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17031634 proc_lsst.multi INFO: [multi] job_id 17031634 2024-03-14 05:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-14 05:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031634 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031634 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17031634 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-14 05:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17031635 proc_lsst.multi INFO: [multi] job_id 17031635 2024-03-14 05:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-14 05:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031635 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031635 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17031635 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-14 05:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17031636 proc_lsst.multi INFO: [multi] job_id 17031636 2024-03-14 05:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-14 05:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031636 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031636 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17031636 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-14 05:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17031637 proc_lsst.multi INFO: [multi] job_id 17031637 2024-03-14 05:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-14 05:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031637 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031637 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17031637 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-14 05:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-14 05:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 17031638 proc_lsst.multi INFO: [multi] job_id 17031638 2024-03-14 05:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-14 05:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031638 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031638 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17031638 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-14 05:33:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-14 05:33:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 05:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 05:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 05:33:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 05:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 05:33:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 05:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 05:33:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 05:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 05:33:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 05:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-14 05:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54602 --result_port=54827 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 05:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 17031639 proc_lsst.multi INFO: [multi] job_id 17031639 2024-03-14 05:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-14 05:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 05:33:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17031639 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17031639 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17031639 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 73 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 153 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd6b6645299108d500d7f5147a65add9 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 200ae794a8a05d487187cd924fcf00fc parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6646fde9-f5b0-4c30-ae0d-962aaea160d1_characterizeImage_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6646fde9-f5b0-4c30-ae0d-962aaea160d1_characterizeImage_947144_5.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/47fcee62-5537-43ca-915e-8744e5be6211_isr_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/47fcee62-5537-43ca-915e-8744e5be6211_isr_947144_5.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 779ecbdf0b34736fee6cb02a91840db4 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 3164362c31b290b07b46637a855e36f0 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/74757817-24a1-451e-a321-b71e8ed8c2a2_characterizeImage_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/74757817-24a1-451e-a321-b71e8ed8c2a2_characterizeImage_947246_15.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c11c871c-3548-4d0c-b44b-75b7475f1808_isr_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c11c871c-3548-4d0c-b44b-75b7475f1808_isr_947246_15.stderr 2024-03-14 05:33:45 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:33:45 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:45 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:45 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:45 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:46 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:46 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:46 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:46 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:47 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:47 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:47 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:48 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:48 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:48 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:48 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:49 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:49 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2ecc734e13c7082ee84ed22f8605fa2 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash f7f4ef0ccaed14b95f83beceb0495062 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bf14c248-2c44-4477-8e26-08d0e0853a5c_characterizeImage_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bf14c248-2c44-4477-8e26-08d0e0853a5c_characterizeImage_947142_19.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c3b95d8-0411-4897-b612-b254b241948c_isr_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c3b95d8-0411-4897-b612-b254b241948c_isr_947142_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:49 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 730547f18c01b9a158e75a329a21a575 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 8868c53df714b25a92b17edc74963295 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/09fc6ca8-a463-4327-a29e-b9d33ee27c47_characterizeImage_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/09fc6ca8-a463-4327-a29e-b9d33ee27c47_characterizeImage_947140_51.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/4bd1659a-b3ac-4e30-a90c-228e37ec07e0_isr_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/4bd1659a-b3ac-4e30-a90c-228e37ec07e0_isr_947140_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:50 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b574bff7f6485341eaa7a2203661f4a1 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 73662b0167b48207b278d9b8e48c382c parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8f603b60-8650-402a-be6c-69346a58dea6_characterizeImage_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8f603b60-8650-402a-be6c-69346a58dea6_characterizeImage_947144_46.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f385f167-3e9c-401b-9e00-ff38f8b09854_isr_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f385f167-3e9c-401b-9e00-ff38f8b09854_isr_947144_46.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 300e7d00cc5ba8796c7755ca60fc37cc with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash ff1a9e901263374d4eb613e228d9e4fb parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/55a3c841-70a9-4f6e-a8c5-0b082e3821b2_characterizeImage_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/55a3c841-70a9-4f6e-a8c5-0b082e3821b2_characterizeImage_947244_29.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e4886bd7-592c-4a09-99ad-c138520958e0_isr_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e4886bd7-592c-4a09-99ad-c138520958e0_isr_947244_29.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a402d30bbd1b8a271d714155f4b6d3b with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 01e0b15c9a72cb9feb10ae28b9142e40 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d12a66bf-ffa0-4e16-81b9-a4bffc36581c_characterizeImage_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d12a66bf-ffa0-4e16-81b9-a4bffc36581c_characterizeImage_947142_5.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/99d1a248-2653-4be9-926b-e003e30d69e5_isr_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/99d1a248-2653-4be9-926b-e003e30d69e5_isr_947142_5.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 693ff734678a2b28dfe26a9c513bf35e with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 18900f346a2aaf707a899c2394bb6ac5 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:50 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:52 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9dd0598a-f882-4e4f-8e38-3c35a85bc954_characterizeImage_947142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9dd0598a-f882-4e4f-8e38-3c35a85bc954_characterizeImage_947142_40.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c3bb43e1-8ab0-4e43-8bab-b8269e0bd725_isr_947142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c3bb43e1-8ab0-4e43-8bab-b8269e0bd725_isr_947142_40.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e04472361a3c3336cf4a794e81104fc with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 65dfe36ba0bd8987d12939d30f79b9d7 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e3c09037-4843-4ad0-b676-d5d020c7a585_characterizeImage_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e3c09037-4843-4ad0-b676-d5d020c7a585_characterizeImage_947246_26.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/85cafeea-0456-46a4-b07b-9b9560127c00_isr_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/85cafeea-0456-46a4-b07b-9b9560127c00_isr_947246_26.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d5b75c3ef10552e8648a8e96904b607 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 64ed862c169e8ab866c05f84ae1d4e3e parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/daeeaa24-c2b5-42c0-9030-bfc306690e48_characterizeImage_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/daeeaa24-c2b5-42c0-9030-bfc306690e48_characterizeImage_947244_52.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fb2331ed-ae68-49fe-941a-63b1cfd74129_isr_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fb2331ed-ae68-49fe-941a-63b1cfd74129_isr_947244_52.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105158ec45c63b9818d89f8c510db39c with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash e4360b5f96a50a3fbfaf603601779ae0 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:54 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ddfba5e5-ed84-4633-8ede-d6e7febfc6d3_characterizeImage_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ddfba5e5-ed84-4633-8ede-d6e7febfc6d3_characterizeImage_947144_8.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/40184bcd-a69c-4f9e-993f-cb37107e4b39_isr_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/40184bcd-a69c-4f9e-993f-cb37107e4b39_isr_947144_8.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5eb55f06b2fec74e0944e550d7975ed with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash e2a199f2b78e233b9613f1dfbcd3cec7 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b5b5cd0d-66f3-46b8-83fb-2caeaabd806e_characterizeImage_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b5b5cd0d-66f3-46b8-83fb-2caeaabd806e_characterizeImage_947142_33.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ec9729ef-7b2d-4ef3-9160-3c6996f6908a_isr_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ec9729ef-7b2d-4ef3-9160-3c6996f6908a_isr_947142_33.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b244351b853a18d67fa342892eae560c with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 2ba35635316bb8ce6abbb3a38b845c41 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/67b1a32f-05a7-4474-a9bf-e8f3379c418e_characterizeImage_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/67b1a32f-05a7-4474-a9bf-e8f3379c418e_characterizeImage_947142_15.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a96086a5-1166-4651-8563-bb94e92ad551_isr_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a96086a5-1166-4651-8563-bb94e92ad551_isr_947142_15.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 395fb26389bd9915d1e2958ec7c36085 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 8aabc2e534c7d7d0738f62ace1b3c41f parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/29862f05-0b28-4d84-ad0f-fca665c39d99_characterizeImage_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/29862f05-0b28-4d84-ad0f-fca665c39d99_characterizeImage_947144_25.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/24451149-6716-4474-8af8-72b1548c345f_isr_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/24451149-6716-4474-8af8-72b1548c345f_isr_947144_25.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a3113e0be2f5d8242470445d7401b33 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash e2158056c973c37e7e8aa3cac10ce520 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5e18a9e4-4238-4f49-95f1-946c59469ab0_characterizeImage_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5e18a9e4-4238-4f49-95f1-946c59469ab0_characterizeImage_947142_14.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/620a61bc-5427-4149-8cfa-a709ceed85e5_isr_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/620a61bc-5427-4149-8cfa-a709ceed85e5_isr_947142_14.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1932ff96274e6679acaa0bcb7a081a4 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash cc1d8718fc97ce0d8fefc42398d2ef3c parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:55 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d8055a3-1679-46d0-8a2d-08ec78e91a58_characterizeImage_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d8055a3-1679-46d0-8a2d-08ec78e91a58_characterizeImage_947140_25.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a0ac78ae-9fb5-4bdf-b96b-5bd0ea6ee137_isr_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a0ac78ae-9fb5-4bdf-b96b-5bd0ea6ee137_isr_947140_25.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdcfb876cfc1955a7918aa736f4e87f6 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e702a0de56c39a5a8b7d6c71305a01ac parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:55 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/79db0a9b-0e67-4ace-aed6-8ef45281cc49_characterizeImage_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/79db0a9b-0e67-4ace-aed6-8ef45281cc49_characterizeImage_947140_49.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e9908082-3f38-4ab3-87e8-66f3ae9ce237_isr_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e9908082-3f38-4ab3-87e8-66f3ae9ce237_isr_947140_49.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fe4180a9a3bee6f19e03b2bf9d94708 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash a9be903b5838c422b91a1dfa4d2a90e3 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:56 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fdb7ec32-dac2-419a-bf47-e804a03f6607_characterizeImage_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fdb7ec32-dac2-419a-bf47-e804a03f6607_characterizeImage_947246_56.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3bc0f4ae-7924-4265-84c4-527f1e6dc4fc_isr_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3bc0f4ae-7924-4265-84c4-527f1e6dc4fc_isr_947246_56.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c50e2fb00e9bbe6a9435f1af8fcf12dd with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash fda88171db8b7e594f90575e22a9b777 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/713019d4-699a-4726-90cc-a08dccaf804a_characterizeImage_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/713019d4-699a-4726-90cc-a08dccaf804a_characterizeImage_947248_17.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/224bf9b4-15ca-4bf5-8775-13dd0d51f094_isr_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/224bf9b4-15ca-4bf5-8775-13dd0d51f094_isr_947248_17.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b79f610408f6985cd058629699c5751c with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 516b9d7897df60df8765f2f2e0f425f6 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c1288e4e-2fec-48c9-855a-6633544f52f0_characterizeImage_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c1288e4e-2fec-48c9-855a-6633544f52f0_characterizeImage_947248_18.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/72d77da5-e096-461a-86e2-e3056c03476f_isr_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/72d77da5-e096-461a-86e2-e3056c03476f_isr_947248_18.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c765031b18745bfc2800198a46928df9 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c0f01eabc45d4c8e1fa24fcccc676a36 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:57 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/8b550919-2929-488c-9c84-494c08a21b8f_characterizeImage_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/8b550919-2929-488c-9c84-494c08a21b8f_characterizeImage_947244_23.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/04703ad5-e5c9-4b14-b3ba-f2744bd00fb7_isr_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/04703ad5-e5c9-4b14-b3ba-f2744bd00fb7_isr_947244_23.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a261cf77cdabd11f7dcbceb60261a52b with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 5635e758c2c796eafb4ede318e42ca71 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/ee0e0f5e-8eef-4522-bbe3-30a2213f6ace_characterizeImage_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/ee0e0f5e-8eef-4522-bbe3-30a2213f6ace_characterizeImage_947248_30.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9fa27c8b-9db4-4e28-ad86-3ad43d46469c_isr_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9fa27c8b-9db4-4e28-ad86-3ad43d46469c_isr_947248_30.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e85baa8c87ae8ba63ad0fc5c6eaafce with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 4f9cf3c4a75d8b27e51b871d40c1d68c parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/8a59f5a3-019d-4c45-9a10-114d251b121c_characterizeImage_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/8a59f5a3-019d-4c45-9a10-114d251b121c_characterizeImage_947246_36.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/fc59b8a2-eb1f-4f77-b14d-f908c67e5430_isr_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/fc59b8a2-eb1f-4f77-b14d-f908c67e5430_isr_947246_36.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f561672e5b4008c54a7ef396997c1284 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash e3e6a980bcc644107b1ced8f454fbe69 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/897a2906-2085-49cd-af33-548a9c936d71_characterizeImage_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/897a2906-2085-49cd-af33-548a9c936d71_characterizeImage_947140_23.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2ca74997-af10-4fcd-9193-f7f85ee55a3e_isr_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2ca74997-af10-4fcd-9193-f7f85ee55a3e_isr_947140_23.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2401ce20a6236d72713be2f91417e19 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash e7f94543ef50b6c4836f385a4f8c5f36 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:58 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bfdd9d8d-d5a2-447c-85f4-97296500fe72_characterizeImage_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bfdd9d8d-d5a2-447c-85f4-97296500fe72_characterizeImage_947248_37.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/54977349-b37b-4b90-a700-b3edb6daabb7_isr_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/54977349-b37b-4b90-a700-b3edb6daabb7_isr_947248_37.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54b7b84bb5c57f3f868af8446c64ddbd with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 1f11f2eee84ec7025f388a75640144e3 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b7d19f2e-374a-426b-b40d-dafcdbb52c20_characterizeImage_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b7d19f2e-374a-426b-b40d-dafcdbb52c20_characterizeImage_947246_47.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ff0cd60a-2f88-40b8-9171-e4337fb775ba_isr_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ff0cd60a-2f88-40b8-9171-e4337fb775ba_isr_947246_47.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1d786293f0a2eee178aae814b85d583 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash aa35657a505c6040e22341e545e288c8 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/619abb6c-3411-46ae-8bf2-be722cce9480_characterizeImage_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/619abb6c-3411-46ae-8bf2-be722cce9480_characterizeImage_947246_48.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/910fdbfa-f81a-4f16-9d05-9776d0410ab3_isr_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/910fdbfa-f81a-4f16-9d05-9776d0410ab3_isr_947246_48.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927c1e39ec74fe258b291efe7517d4f6 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash b3b5dafe718782c58435eabbb113e728 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/7e7d85bc-1963-498b-91c5-7bba76a863fa_characterizeImage_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/7e7d85bc-1963-498b-91c5-7bba76a863fa_characterizeImage_947244_51.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4477e566-f5de-46b2-84e2-a16c831962e9_isr_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4477e566-f5de-46b2-84e2-a16c831962e9_isr_947244_51.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b1478262c01060c10fd2179cfb914f with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 65f2ef4cc9b32054447c0ef6f647e080 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a3bc7-0622-423d-bff4-42673c4b2194_characterizeImage_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a3bc7-0622-423d-bff4-42673c4b2194_characterizeImage_947248_5.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47ef208d-2963-4a76-89cb-91a82513b4a3_isr_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47ef208d-2963-4a76-89cb-91a82513b4a3_isr_947248_5.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ed09dc1a99ed29896d1083613966e6 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 21f097abae778be0610a2aed3e7ffdfe parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9843b2f3-0521-4bc5-a294-57d89da6482c_characterizeImage_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9843b2f3-0521-4bc5-a294-57d89da6482c_characterizeImage_947142_41.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/d6ac74e1-d5d7-46bd-8548-f8933cbaba77_isr_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/d6ac74e1-d5d7-46bd-8548-f8933cbaba77_isr_947142_41.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ad3193883b0bcdc8a6b22efe07fa525 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 6866af8c263a1d51fe5d21dda1f762b9 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a94db-f10d-4346-896b-a063f981be2b_characterizeImage_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a94db-f10d-4346-896b-a063f981be2b_characterizeImage_947248_48.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7c0f7f9e-4c2b-4ecb-ba3e-27ba6c154212_isr_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7c0f7f9e-4c2b-4ecb-ba3e-27ba6c154212_isr_947248_48.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67148fcb746b11d217ee0957295bb631 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash d163078f0c56c810c1188eb8e365f779 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:58 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2220a06e-6ce7-4f80-abc9-03adae6bdb00_characterizeImage_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2220a06e-6ce7-4f80-abc9-03adae6bdb00_characterizeImage_947140_10.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/89ebfa08-2e40-4d1f-9139-c68dc61a3f42_isr_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/89ebfa08-2e40-4d1f-9139-c68dc61a3f42_isr_947140_10.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe9fc52eef7fa81e0813285aa5a90f40 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 910563245a28e6c4f4d71f799bd14438 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a9e887e2-3d61-4aa3-9090-15cb2ba3e1c6_characterizeImage_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a9e887e2-3d61-4aa3-9090-15cb2ba3e1c6_characterizeImage_947248_45.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/42eb66b8-2feb-4be2-b05d-ddc96bfd9976_isr_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/42eb66b8-2feb-4be2-b05d-ddc96bfd9976_isr_947248_45.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e9b24d5125cad3392c033640b0d91e with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash af85ecb0569d21f6565c23f1d9d40975 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/190a557c-37ab-445a-8672-b55b02b07267_characterizeImage_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/190a557c-37ab-445a-8672-b55b02b07267_characterizeImage_947144_18.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a0890661-8dfa-48fa-9dd5-628e40ac3887_isr_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a0890661-8dfa-48fa-9dd5-628e40ac3887_isr_947144_18.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91dcf2c0603c19db04204febc809edf with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 481b88b37d8db89083f7f5da9965461c parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f9631638-6f4b-48c6-8941-c1faab0a08b9_characterizeImage_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f9631638-6f4b-48c6-8941-c1faab0a08b9_characterizeImage_947144_35.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2416275a-d508-4484-9b5f-11daf56a4449_isr_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2416275a-d508-4484-9b5f-11daf56a4449_isr_947144_35.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38bd2b0f7e0537685ae5e7b0f32e2e9 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 219b969594c2abfb1b9fe62bcb06ee11 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c1f3d56f-ed39-4bbb-900d-7798f73af8a4_characterizeImage_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c1f3d56f-ed39-4bbb-900d-7798f73af8a4_characterizeImage_947140_13.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c2f3559d-5253-4f06-b3f1-fbfed9cf0020_isr_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c2f3559d-5253-4f06-b3f1-fbfed9cf0020_isr_947140_13.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f9aaafc9aa4771dd39a36641699141 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash b0f477df6f271645fc4e951d9d808ec8 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/84e351fb-9e2d-498a-bd11-9d13cbf51373_characterizeImage_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/84e351fb-9e2d-498a-bd11-9d13cbf51373_characterizeImage_947140_30.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b575cbea-fbc0-48b6-8fa3-70c8c20c92c8_isr_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b575cbea-fbc0-48b6-8fa3-70c8c20c92c8_isr_947140_30.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 150ab346d487c65cdab20e45d90fb29a with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash e9b2284aa93b6254f04f063bfcbad753 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:58 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a7690d4-da6d-404d-933a-cff913608f8d_characterizeImage_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a7690d4-da6d-404d-933a-cff913608f8d_characterizeImage_947244_9.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/34fc6a6e-3eaf-4007-a94f-4d9199cd4065_isr_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/34fc6a6e-3eaf-4007-a94f-4d9199cd4065_isr_947244_9.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30b2fa6dcb7ebfca3f02a829c13e5ea5 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 8dce7cc53aca45f742a36546ead0aedf parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9b525d0f-0755-4e4f-ba15-494e3219edb9_characterizeImage_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9b525d0f-0755-4e4f-ba15-494e3219edb9_characterizeImage_947244_56.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3e4a6f7-99c6-4110-917b-5be5ee87e7d8_isr_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3e4a6f7-99c6-4110-917b-5be5ee87e7d8_isr_947244_56.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86fc659236db8d1a73c8433e035a3a5b with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 623a7f260cf718d6a205fe8812286eab parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6066e577-1b11-4254-bb73-58de772a61d4_characterizeImage_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6066e577-1b11-4254-bb73-58de772a61d4_characterizeImage_947140_57.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5101d044-ca79-45cc-9026-639e35e06239_isr_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5101d044-ca79-45cc-9026-639e35e06239_isr_947140_57.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 555d1859c7c4d0d487b52a877ecb72f1 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 65da0795a1b6ca9ccbe2c4eb7dfda63d parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a3a9602f-c1fc-4cbe-8ed0-c4632b834490_characterizeImage_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a3a9602f-c1fc-4cbe-8ed0-c4632b834490_characterizeImage_947140_43.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/7c587559-5e7e-445e-8c71-0b265507e57c_isr_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/7c587559-5e7e-445e-8c71-0b265507e57c_isr_947140_43.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff76a99034e7022741d9f587f9c32f5b with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 3d44256025e07c2ef3c57110864f69e7 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/fbdab855-0ed3-44a9-9ad9-93435cf13e82_characterizeImage_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/fbdab855-0ed3-44a9-9ad9-93435cf13e82_characterizeImage_947142_58.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/abe4ab44-2748-46df-a77d-73d1f0df9178_isr_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/abe4ab44-2748-46df-a77d-73d1f0df9178_isr_947142_58.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 220a37f6484c0c43d29d7e2b6c07f3a7 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 84d62c331594658beec2dd8e3c145ef3 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5f2807f2-9671-4181-a68e-cdcf5b94d0d5_characterizeImage_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5f2807f2-9671-4181-a68e-cdcf5b94d0d5_characterizeImage_947246_60.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/7b30d795-3127-4fea-b848-8ed806e67908_isr_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/7b30d795-3127-4fea-b848-8ed806e67908_isr_947246_60.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d03b5dee94628f38c0a31f84f195577 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash bf99c1e847d3fccc5fb91e6f18040f61 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/2b090fca-c0c7-499c-add8-f8df959bd4a3_isr_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/2b090fca-c0c7-499c-add8-f8df959bd4a3_isr_947244_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f6c754fb566a127a7b8569dd5e10d5f with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash cf11ece3f3c4beb0c1ddbd8084b00923 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/0992cadd-b0f5-4f8b-94c9-a6978beebbc7_characterizeImage_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/0992cadd-b0f5-4f8b-94c9-a6978beebbc7_characterizeImage_947142_28.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af648991-5ceb-40dd-9508-64eddaa77a62_isr_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af648991-5ceb-40dd-9508-64eddaa77a62_isr_947142_28.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c8033b71c8b11ac2958de451d7393b with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash ec364434290312ede7f3aabb627fa4a1 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/447bdd8c-3efa-4525-b5ea-7041a95ec749_characterizeImage_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/447bdd8c-3efa-4525-b5ea-7041a95ec749_characterizeImage_947142_45.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/0722760b-f4a1-4cbc-9e7e-139af04c72d9_isr_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/0722760b-f4a1-4cbc-9e7e-139af04c72d9_isr_947142_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:33:59 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe638c953e21cc456ee50a5c357d58cf with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 4762a4a00bf9868aebdb9647d6368465 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/134f01ae-197d-4d48-a9c9-ae64fe9dba33_characterizeImage_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/134f01ae-197d-4d48-a9c9-ae64fe9dba33_characterizeImage_947246_11.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0527d6ed-3aac-4b82-8c9d-79e240fcad0b_isr_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0527d6ed-3aac-4b82-8c9d-79e240fcad0b_isr_947246_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ce09466f83d8431652637570bb6fed with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 128e35c5fb0ed7657c9d1630979439ba parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7637263a-3c55-4029-a92f-26f8c9639beb_characterizeImage_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7637263a-3c55-4029-a92f-26f8c9639beb_characterizeImage_947246_45.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f802199d-553e-4e23-adbf-9b9fdf1aa81f_isr_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f802199d-553e-4e23-adbf-9b9fdf1aa81f_isr_947246_45.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06bc3f6306b65ae779a2a17512c18290 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 956d93133343d88e1c334ce2ffcc22cf parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/eb04b945-9168-46a3-a094-705a4ac23b44_characterizeImage_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/eb04b945-9168-46a3-a094-705a4ac23b44_characterizeImage_947144_23.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d8bc9764-e6e4-4b46-8434-96794cd5ee6c_isr_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d8bc9764-e6e4-4b46-8434-96794cd5ee6c_isr_947144_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4edd573fa74b864a5e6cec82025c090 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 471c3c4dcf36fee6f65b5d4d3097c6c0 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c52c15dd-7eba-4d55-94f3-36eeca94fa83_characterizeImage_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c52c15dd-7eba-4d55-94f3-36eeca94fa83_characterizeImage_947248_61.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1052041b-13de-4f35-9bac-22a32765afeb_isr_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1052041b-13de-4f35-9bac-22a32765afeb_isr_947248_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:00 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392de27de084303111849902cdf9900c with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash c5fe5e23b2a56412d05e7ea5820f3a0b parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:01 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/7ca0aff0-e93f-403f-9e85-6e05c5f04728_characterizeImage_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/7ca0aff0-e93f-403f-9e85-6e05c5f04728_characterizeImage_947142_3.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/87818e9d-5e30-4a2d-af51-7d9d1c529780_isr_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/87818e9d-5e30-4a2d-af51-7d9d1c529780_isr_947142_3.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e849cd0d1a998e389b33785a12ce5df5 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 0b0ed97d4098fd5781c6395b1389d240 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5144c21e-d6fc-4b85-96da-ed5f3ced8bdf_characterizeImage_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5144c21e-d6fc-4b85-96da-ed5f3ced8bdf_characterizeImage_947248_4.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5247d168-3387-4178-93ff-a3fcb5d8b5b3_isr_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5247d168-3387-4178-93ff-a3fcb5d8b5b3_isr_947248_4.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 110a340d50b4005bee3d8908fa9adcb0 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 07f4c2740cd249ddca9240414dc33f68 parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:01 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/6db06895-ef6b-452d-86c3-9c2bb63c08d3_characterizeImage_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/6db06895-ef6b-452d-86c3-9c2bb63c08d3_characterizeImage_947248_60.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/89daa40f-c4df-4c52-bc47-caee0aa16ec8_isr_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/89daa40f-c4df-4c52-bc47-caee0aa16ec8_isr_947248_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:01 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:01 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5cd25b596d7c68bc8e20e5401e2d688 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 3de2acaab3f27980fcb77c1dab8a9cd3 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4e504a77-fb5f-4498-87b7-9198b7dc37c0_characterizeImage_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4e504a77-fb5f-4498-87b7-9198b7dc37c0_characterizeImage_947144_54.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e8613c21-f5fb-4618-bd8d-f9168a17e893_isr_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e8613c21-f5fb-4618-bd8d-f9168a17e893_isr_947144_54.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ae5769a28bee0f940c49cf7b433a188 with result from task 222 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:02 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 4cdd0b4cca9401b7bc265906f3b50835 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:02 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:21 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e76f64e-e25e-4937-84bd-ee73f3785faa_characterizeImage_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e76f64e-e25e-4937-84bd-ee73f3785faa_characterizeImage_947246_55.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/63a1735f-1762-467d-b923-bd7c018ce5a0_isr_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/63a1735f-1762-467d-b923-bd7c018ce5a0_isr_947246_55.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb340cc292b2018f00e1666bb33bb0c7 with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash a613e1846c4113a251f443a39045fbfd parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67faac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/664c40a8-690c-4653-8cf8-ce21b1f35af1_characterizeImage_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/664c40a8-690c-4653-8cf8-ce21b1f35af1_characterizeImage_947144_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/66d2df00-c846-4b54-a324-33fb0b2e74f4_isr_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/66d2df00-c846-4b54-a324-33fb0b2e74f4_isr_947144_21.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4812fdb01a50fd0793fb5ce292a48cf with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 1a7050a4d731f912cb8615601fcee24f parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/13215e26-02be-4c66-abb2-2f05c62e3c98_isr_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/13215e26-02be-4c66-abb2-2f05c62e3c98_isr_947140_2.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef1dcf6cc1923f8195798d865b71047 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 8fa0a2bb959977f4c385b8c766b226c6 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:27 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:36 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e507c35b-c1b5-4118-8b38-114c00fee7e9_characterizeImage_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e507c35b-c1b5-4118-8b38-114c00fee7e9_characterizeImage_947244_34.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f32e2a0e-cf55-49fd-bc33-293fc36b590b_isr_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f32e2a0e-cf55-49fd-bc33-293fc36b590b_isr_947244_34.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a33b67d4ae9b68e96d42c4f25cc985b with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 0385a24d42fd4ea527d0d129a997d67c parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a945e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93e77fbd-abdb-4eff-bec5-b18ce4e61ae2_characterizeImage_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93e77fbd-abdb-4eff-bec5-b18ce4e61ae2_characterizeImage_947244_1.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/21469f7e-734f-473e-bc95-82514305d6c1_isr_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/21469f7e-734f-473e-bc95-82514305d6c1_isr_947244_1.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8dcb041fc28025d2033bc42c006282 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash e1aa76fc976e759bb6925a40adeaac7e parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:52 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/20d9b811-a6ed-4b76-86b3-b9ba15826cfa_isr_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/20d9b811-a6ed-4b76-86b3-b9ba15826cfa_isr_947142_7.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0145d1be226e6eb70a42019948010fe8 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash e96a1f72fab3ef854b9f966a7d557af3 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/91a9f55d-1562-4aa3-89cd-7e84d349b732_characterizeImage_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/91a9f55d-1562-4aa3-89cd-7e84d349b732_characterizeImage_947144_49.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/55772afd-eafb-4bb2-ba40-70abc9989315_isr_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/55772afd-eafb-4bb2-ba40-70abc9989315_isr_947144_49.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39cbb6472d45f9ced8052eb0de8a8cb1 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash f90fc6c73dd74cf36a72d3709c012fec parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a958a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:57 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a843efa3-956c-4212-aa48-0d2318562846_characterizeImage_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a843efa3-956c-4212-aa48-0d2318562846_characterizeImage_947142_37.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/93dc7252-c908-4333-90da-5ba0d6625f20_isr_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/93dc7252-c908-4333-90da-5ba0d6625f20_isr_947142_37.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f39d76499da9312e12719eb4ba7c8d with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 57f0b525918663002efc531ff1367fc7 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/ac3ca6f0-dfb0-462b-b74d-c8dd9f057d72_characterizeImage_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/ac3ca6f0-dfb0-462b-b74d-c8dd9f057d72_characterizeImage_947140_50.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e10874e4-8b3f-43ab-9151-948d682a5352_isr_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e10874e4-8b3f-43ab-9151-948d682a5352_isr_947140_50.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3c0d4d1d36f9cfeab5797335bd0248 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 2df6de072637b22dafe640636cd1424e parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:58 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/86fd2a0e-8c12-4c59-9322-fea161faf37e_characterizeImage_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/86fd2a0e-8c12-4c59-9322-fea161faf37e_characterizeImage_947144_32.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5948332e-b177-4108-a2df-acfa044b60a8_isr_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5948332e-b177-4108-a2df-acfa044b60a8_isr_947144_32.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6958aca661423fd284a61ea4446371a with result from task 114 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:58 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 7a75d96a449b757a6cd3b44539ecee7b parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/29384594-101d-4e4f-bb53-579c6f8428f4_characterizeImage_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/29384594-101d-4e4f-bb53-579c6f8428f4_characterizeImage_947244_48.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63da5071-71ac-416d-accc-cfdde6187d39_isr_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63da5071-71ac-416d-accc-cfdde6187d39_isr_947244_48.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef16fe97380bc589a328ddb17313a0a with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 08b3b875ed5a16e91ede7841fe69ae55 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:59 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/286d45ba-8e3a-471c-9d83-f5d89c4cf5d7_characterizeImage_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/286d45ba-8e3a-471c-9d83-f5d89c4cf5d7_characterizeImage_947248_7.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/b3d22522-5531-4b87-9c48-bd510b9c4c9c_isr_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/b3d22522-5531-4b87-9c48-bd510b9c4c9c_isr_947248_7.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 463d405545311fc64b63340365f7ec5a with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash cba7f09399bb4e3ef1927a41e269c82c parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/13725f80-059f-48c4-aabc-100858ee3166_characterizeImage_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/13725f80-059f-48c4-aabc-100858ee3166_characterizeImage_947142_47.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/80ac9a0d-c504-4bdd-8b53-15c3e8e8a1ec_isr_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/80ac9a0d-c504-4bdd-8b53-15c3e8e8a1ec_isr_947142_47.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe263cac7121f3140d1c8c1133da7cdb with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash c0549628f9b84ad4f92a29a465943e8d parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/28596b8a-446a-42de-ab51-24faeef78703_characterizeImage_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/28596b8a-446a-42de-ab51-24faeef78703_characterizeImage_947246_33.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/9230dea2-fee4-45eb-a428-dfbbd105e953_isr_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/9230dea2-fee4-45eb-a428-dfbbd105e953_isr_947246_33.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2216d39185b853c11ebc49f84d779d with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 14a54e457700d67da95927b3d8ab9021 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cc4a68a2-84c8-4a9f-b7d4-fbbcba82071f_characterizeImage_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cc4a68a2-84c8-4a9f-b7d4-fbbcba82071f_characterizeImage_947248_1.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8a3bb6ca-2dfe-4c46-9bb6-f45ef1da6691_isr_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8a3bb6ca-2dfe-4c46-9bb6-f45ef1da6691_isr_947248_1.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550eb4c910474a66865d65817b2fcff8 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 51d2dbf8e3444c4500d09611825872c1 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:59 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/71215b4d-9ef8-4e36-8863-344e0406da84_characterizeImage_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/71215b4d-9ef8-4e36-8863-344e0406da84_characterizeImage_947142_62.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/9228cd52-24df-4bf6-84b1-4aaaee21616d_isr_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/9228cd52-24df-4bf6-84b1-4aaaee21616d_isr_947142_62.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e21c3b88305376e6c7349a7f51901539 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 15ecd42793d43aaa5e82231680c692e2 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:59 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3f94fc11-c0c0-48fa-b8ad-ce0f6d42915e_isr_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3f94fc11-c0c0-48fa-b8ad-ce0f6d42915e_isr_947248_2.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69dbcff40d3e56c6d7274bc106c7f5c9 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 6a9e9dd989bc1cdf5523a9522dffd35c parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a976a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/93e67bda-7892-4456-9683-fe12a48155c3_characterizeImage_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/93e67bda-7892-4456-9683-fe12a48155c3_characterizeImage_947144_6.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f681b0b2-1681-4e7e-9fe5-152594088514_isr_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f681b0b2-1681-4e7e-9fe5-152594088514_isr_947144_6.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed290496e1cde05ea1544739f48e8619 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 5a726b75e81596e12d1569a4b49de124 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a974c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/80ba1e19-612f-49c8-adc7-c4aeab104cbf_characterizeImage_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/80ba1e19-612f-49c8-adc7-c4aeab104cbf_characterizeImage_947248_56.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a927c3b3-26d3-43e1-b489-23b7be1319ef_isr_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a927c3b3-26d3-43e1-b489-23b7be1319ef_isr_947248_56.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f955f18c801602853f3abfc4620faaa with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash af6af9ab30c4b07d67eb9a0027558b01 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5dcce05a-dace-419e-ae9b-3934a548f416_characterizeImage_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5dcce05a-dace-419e-ae9b-3934a548f416_characterizeImage_947248_44.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/22f542b1-f5a2-4093-9255-b12915487c7e_isr_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/22f542b1-f5a2-4093-9255-b12915487c7e_isr_947248_44.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37bfbe09515f3dfde84ff8aec1464b26 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 4c19930110d2674e03b167e6a8ec2eec parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/9b8d0f33-2eb3-4a66-a492-1550e40c8a60_characterizeImage_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/9b8d0f33-2eb3-4a66-a492-1550e40c8a60_characterizeImage_947144_51.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2f872bdd-de3f-444c-ae56-6b159c163ec0_isr_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2f872bdd-de3f-444c-ae56-6b159c163ec0_isr_947144_51.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be57bc287d6361c0a1d4c016354093e with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 7f3f93601d47b573e6e91d0ccff86a76 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0fc72a20-1653-4dec-a335-319c50cd598b_characterizeImage_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0fc72a20-1653-4dec-a335-319c50cd598b_characterizeImage_947246_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f2e218e4-78e2-4e46-9416-45fea67f0544_isr_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f2e218e4-78e2-4e46-9416-45fea67f0544_isr_947246_16.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8784016a1d36d6da19b5eb1baaefeec with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash 5f2d42e249424cd560b13fe3f83f2f3f parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/38a8ec16-be99-41b8-ada0-e2efd5f1accd_characterizeImage_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/38a8ec16-be99-41b8-ada0-e2efd5f1accd_characterizeImage_947142_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7e84777d-7664-45c6-b2ed-daf6e18be943_isr_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7e84777d-7664-45c6-b2ed-daf6e18be943_isr_947142_31.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a1c79813ab6e5f5df6472c5696a33fa with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash c7bc8d9cb95615e8356cf1a445b1cfea parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a433c596-319e-4c01-91dc-aff51bc32733_characterizeImage_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a433c596-319e-4c01-91dc-aff51bc32733_characterizeImage_947142_59.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/afbd734b-02ed-42ee-a597-26127823418a_isr_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/afbd734b-02ed-42ee-a597-26127823418a_isr_947142_59.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc6c16f55ae615bec28ea3d6ac4932a3 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 7b70a4fb64b659932fb5efeeeaa1e30f parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/be35fd34-fea3-4941-a91c-c8a8dc4188d0_characterizeImage_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/be35fd34-fea3-4941-a91c-c8a8dc4188d0_characterizeImage_947142_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b6017c66-c90e-45e9-b10e-c7a45ab7201d_isr_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b6017c66-c90e-45e9-b10e-c7a45ab7201d_isr_947142_46.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf7efc556929f66c005227f2441cc469 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash e3adf52d0e8a53e316bb44fa78037263 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/492b2a4c-afb0-47d7-b986-c594ba94cc1f_characterizeImage_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/492b2a4c-afb0-47d7-b986-c594ba94cc1f_characterizeImage_947142_55.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1c4b1728-bce9-4551-a1f2-36923eeee69f_isr_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1c4b1728-bce9-4551-a1f2-36923eeee69f_isr_947142_55.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1c011d570423e4f0f8d36499bcfcc03 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash d395992b81eaed01272177dd7dce81e3 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:59 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/576c0279-80a9-4236-b566-964df7b6e51d_characterizeImage_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/576c0279-80a9-4236-b566-964df7b6e51d_characterizeImage_947244_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f135a2aa-c7c7-4e7d-ba96-2a8e12b8a38d_isr_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f135a2aa-c7c7-4e7d-ba96-2a8e12b8a38d_isr_947244_8.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a58b393db5edb66a2984c7e6ee62588 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash f91a714fac7894bc7a06db902087cd58 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:34:59 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a7ea58cf-9bd0-40d0-8fcf-17bbacabda1d_characterizeImage_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a7ea58cf-9bd0-40d0-8fcf-17bbacabda1d_characterizeImage_947144_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/db86b8a3-94c4-4cb7-a8c0-04fa9e7ba9b2_isr_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/db86b8a3-94c4-4cb7-a8c0-04fa9e7ba9b2_isr_947144_53.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4c68c9242db8ec7ac449a8382f8723 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash e0ae1c5b42d975a5188636dc2386ed91 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/339282ea-4fdb-46ca-8abf-31ffa5bfc015_characterizeImage_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/339282ea-4fdb-46ca-8abf-31ffa5bfc015_characterizeImage_947142_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/610a2081-e60e-409f-acbd-ced7ac3e0a89_isr_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/610a2081-e60e-409f-acbd-ced7ac3e0a89_isr_947142_30.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b184311414e4733f77cd325d72804d65 with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash cabd34c9c32a9403ecac05435491e6dc parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/983f2213-7d03-416b-88cb-381375d587f5_characterizeImage_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/983f2213-7d03-416b-88cb-381375d587f5_characterizeImage_947140_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c52e4deb-4336-4b7f-be81-10c2010ce3a2_isr_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c52e4deb-4336-4b7f-be81-10c2010ce3a2_isr_947140_8.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b2ec95f609c09607e5137be3e04f85 with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash a3fd0125dc601a95c43f597fa7477ddf parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/848f30a8-a75d-4479-b047-cae64c51d4e2_characterizeImage_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/848f30a8-a75d-4479-b047-cae64c51d4e2_characterizeImage_947140_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/db42beb9-ae33-4596-878f-b16083d7b2b7_isr_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/db42beb9-ae33-4596-878f-b16083d7b2b7_isr_947140_47.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1c975c814aae5d92a8557565b0da52c with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 0bac608125216be95a73c31164f4a852 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0addc5f3-d625-40ab-8b6f-2a94797be963_characterizeImage_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0addc5f3-d625-40ab-8b6f-2a94797be963_characterizeImage_947144_33.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b61d2b7c-23aa-46de-a81e-fe2fca5a02f8_isr_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b61d2b7c-23aa-46de-a81e-fe2fca5a02f8_isr_947144_33.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5154ccd9242e1b0cb7154a014203cef9 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 80ffb2c5705a2c779ef69d6482b3eac1 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/41afc57b-552b-40a3-9f98-21e6dfc8bc6f_characterizeImage_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/41afc57b-552b-40a3-9f98-21e6dfc8bc6f_characterizeImage_947246_6.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f0e748f4-2c6b-4f0d-a2c0-a68963d84157_isr_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/f0e748f4-2c6b-4f0d-a2c0-a68963d84157_isr_947246_6.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d6d2afc5ee7a67e6e93bb69c008371 with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 1a436c7581e8283588e7e9cc4fd7a7d2 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/da0829b7-616d-4c79-aa59-7a0b1e4fc079_characterizeImage_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/da0829b7-616d-4c79-aa59-7a0b1e4fc079_characterizeImage_947246_14.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3ad558dd-c907-403b-9694-35182f26cdda_isr_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3ad558dd-c907-403b-9694-35182f26cdda_isr_947246_14.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8cfffc3f04de8ee3750e3453724d42e with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash eb5cb0f9b612cbbc7bd14b3bf29b527e parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5de5eb78-cc3a-4a53-9809-4a06c9a5de0c_characterizeImage_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5de5eb78-cc3a-4a53-9809-4a06c9a5de0c_characterizeImage_947246_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/48c79386-942d-42ce-9789-6af3a292e133_isr_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/48c79386-942d-42ce-9789-6af3a292e133_isr_947246_9.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b865e878a042766a0634eb0e17efbad with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash fc86d3a65cef6383a6bad7218fd997a1 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a3760520-476b-42cb-b17e-54d3e5a76047_characterizeImage_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a3760520-476b-42cb-b17e-54d3e5a76047_characterizeImage_947248_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/601098ca-6a6e-46ca-bb16-a530b21ab962_isr_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/601098ca-6a6e-46ca-bb16-a530b21ab962_isr_947248_39.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919a7aa10347d92e9c4631d570a2c7d6 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 6a09860ed146ea1a56eb8e26f55a0be3 parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/882ca2d2-d8b7-4eab-8b73-a8c6244175e1_characterizeImage_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/882ca2d2-d8b7-4eab-8b73-a8c6244175e1_characterizeImage_947246_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/661c6954-cfd0-41a3-89fb-6dec652eb296_isr_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/661c6954-cfd0-41a3-89fb-6dec652eb296_isr_947246_43.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bcb0dca734f9cb62d99eb7a7c646b4a with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 3b211247b6289953e5beb393a03d69cf parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/bdc87192-98c0-4861-b886-d1e156948ab4_characterizeImage_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/bdc87192-98c0-4861-b886-d1e156948ab4_characterizeImage_947244_24.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7de9b47e-4f25-4066-ba95-f0cad65aaec0_isr_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7de9b47e-4f25-4066-ba95-f0cad65aaec0_isr_947244_24.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dabcdd35bb2591821b1d50a79676f670 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 7746d9e26baf4d531fcb8d45472ba8cb parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/04bfe95b-2beb-4658-9869-60e949ab4f75_characterizeImage_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/04bfe95b-2beb-4658-9869-60e949ab4f75_characterizeImage_947144_24.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/ae5729af-e8d0-47a6-a0f3-1b08ae309c9b_isr_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/ae5729af-e8d0-47a6-a0f3-1b08ae309c9b_isr_947144_24.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc0d4f128780c052c48a61e574011148 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 3a846362b1eeb1659df404760c67fc8f parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8e79bf7c-2d20-4f13-9723-23649a3a41a6_characterizeImage_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8e79bf7c-2d20-4f13-9723-23649a3a41a6_characterizeImage_947144_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/30b7765d-3ed9-486d-ad52-3a8be984ec58_isr_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/30b7765d-3ed9-486d-ad52-3a8be984ec58_isr_947144_31.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fbd62544e81e9b9e41599a729bf863 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 6c1acfe2efce405460bc7d7153010d8a parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0286c90f-d9ae-49d7-9c53-09461a5dbf25_characterizeImage_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0286c90f-d9ae-49d7-9c53-09461a5dbf25_characterizeImage_947244_35.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/65d29a7c-2cf1-456b-9bc0-45f8f478f67f_isr_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/65d29a7c-2cf1-456b-9bc0-45f8f478f67f_isr_947244_35.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf23303f2e56cb9456f11168d8bdf587 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash f54e6e9b4f5995b782e3a2e08ec45a47 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9b698916-78d2-42b6-a4c6-cc5d6f2831c5_characterizeImage_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9b698916-78d2-42b6-a4c6-cc5d6f2831c5_characterizeImage_947246_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a645c62f-42b4-4d5b-988d-7f0eafd09ccf_isr_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/a645c62f-42b4-4d5b-988d-7f0eafd09ccf_isr_947246_22.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e743039ae89c836a608720aaa0178a9d with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 30530524abb1d72baabcce1c80542868 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2b6c577e-bcb0-4d3f-bfec-6691a3ad0d3d_characterizeImage_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2b6c577e-bcb0-4d3f-bfec-6691a3ad0d3d_characterizeImage_947140_19.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e66e8e96-6f93-4e43-ad0d-fc1e89b5e12d_isr_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e66e8e96-6f93-4e43-ad0d-fc1e89b5e12d_isr_947140_19.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d42653ed4d819490f3737faf9517ea6c with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 0776ea8efa22d9769fa7cf8d5058a8aa parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cb451032-49f6-458f-96ed-b7b3b5a96e86_characterizeImage_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cb451032-49f6-458f-96ed-b7b3b5a96e86_characterizeImage_947140_62.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a4cff87f-9da1-44ee-99f7-0e7a17ec1262_isr_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a4cff87f-9da1-44ee-99f7-0e7a17ec1262_isr_947140_62.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2379fe0c9fa15cc62e2b727144f445 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash f2984ef128339f7468cc3f14beefb740 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/92d3e6cd-2b93-4ecc-a110-9c8857b9da9f_characterizeImage_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/92d3e6cd-2b93-4ecc-a110-9c8857b9da9f_characterizeImage_947140_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a5ec563d-821c-4bf7-9472-193076c61fd1_isr_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a5ec563d-821c-4bf7-9472-193076c61fd1_isr_947140_52.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b7cd4aa554258b7920db84e417aa00 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 8e1ddc96156727eedd8a09f5c4927d36 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/000528ed-8c82-419c-9d00-ab5d0adacf15_characterizeImage_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/000528ed-8c82-419c-9d00-ab5d0adacf15_characterizeImage_947244_6.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/9ea6e8f0-1083-4ab6-b719-d679b313c40d_isr_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/9ea6e8f0-1083-4ab6-b719-d679b313c40d_isr_947244_6.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7889e0aae30f3ce417299567ac185db7 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 4ebe723b6cc7367ecdea42521cecbf80 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/593d8d58-118e-4bea-8520-7e2fe26e7273_characterizeImage_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/593d8d58-118e-4bea-8520-7e2fe26e7273_characterizeImage_947142_34.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/61ac4532-f78a-435c-8d3a-8f4a81387083_isr_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/61ac4532-f78a-435c-8d3a-8f4a81387083_isr_947142_34.stderr parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2975a5f0bdb9a74dd0e1bb06d2f5da with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 0b04deb76d29c718c20bac9349d16534 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/af6dd874-0464-44e7-9cd4-37cbea3e76bf_characterizeImage_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/af6dd874-0464-44e7-9cd4-37cbea3e76bf_characterizeImage_947140_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/07110e11-eb46-41b5-9670-28ac6a9faa35_isr_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/07110e11-eb46-41b5-9670-28ac6a9faa35_isr_947140_45.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20a2db64e3f7f3ddb02a3bd0ec0a2d1 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 05ce470fa77f45dd925a2631cc8ee895 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6636160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/27d47967-c3cf-4256-8970-7188f57e351a_characterizeImage_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/27d47967-c3cf-4256-8970-7188f57e351a_characterizeImage_947244_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4d43a621-5613-4af5-a413-d92de164a031_isr_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4d43a621-5613-4af5-a413-d92de164a031_isr_947244_53.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed6eddb1aab68478bec85ce6849dccd with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 8bf64fe0fb13b3665ee1fd8cd1dd5ba9 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4875527b-c883-412b-98d6-db04a8d6f58b_characterizeImage_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4875527b-c883-412b-98d6-db04a8d6f58b_characterizeImage_947248_15.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/76a2d01a-77c5-484d-94ae-e9aa2bfdf0ee_isr_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/76a2d01a-77c5-484d-94ae-e9aa2bfdf0ee_isr_947248_15.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9f5afd7a5fb60c49f812d8ee0f40c53 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash a3fd21ea6f22e8101cf627b0fb7b795b parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7d4c6ed6-451f-4fc4-82bf-0b2a40976805_characterizeImage_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7d4c6ed6-451f-4fc4-82bf-0b2a40976805_characterizeImage_947248_14.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/f5958c32-0d84-4b8f-b241-4ee3e0cbdea2_isr_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/f5958c32-0d84-4b8f-b241-4ee3e0cbdea2_isr_947248_14.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a9e97644e9f6974430d3e226eac222 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash e316351d402ac61eda30b80a88c1f0b3 parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/db2665cc-f1ff-4c49-bef5-cdae5f473a34_characterizeImage_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/db2665cc-f1ff-4c49-bef5-cdae5f473a34_characterizeImage_947144_7.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/382227f5-ca6f-4989-a371-5c581be30b7c_isr_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/382227f5-ca6f-4989-a371-5c581be30b7c_isr_947144_7.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f456f52e30d46e4bfad943dd2e6e76 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash e96f85582a0a556aa7b041ba026911c6 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c575c4f8-3247-4580-b26e-6ec7319984db_characterizeImage_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c575c4f8-3247-4580-b26e-6ec7319984db_characterizeImage_947140_6.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/8b241efc-9124-4d9b-baab-8ccec97df670_isr_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/8b241efc-9124-4d9b-baab-8ccec97df670_isr_947140_6.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11c01d784c843cd318013a0030125326 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 68dea4276cec3b4addc10115c1940c6c parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/56b9f0f0-70db-4ce9-ad6d-aef6b5b65060_characterizeImage_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/56b9f0f0-70db-4ce9-ad6d-aef6b5b65060_characterizeImage_947140_27.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0337d906-7b56-4839-8651-3aaa06f3f9b2_isr_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0337d906-7b56-4839-8651-3aaa06f3f9b2_isr_947140_27.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0712b107ca36e2a3883727eb0456827 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 24f9c01fb7174e2183bee030c732900c parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f36d4638-fea1-472d-bcf4-79f20b4c0661_characterizeImage_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f36d4638-fea1-472d-bcf4-79f20b4c0661_characterizeImage_947248_26.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4cce8c2b-2689-4d3e-92ef-0c5f1e1b215a_isr_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4cce8c2b-2689-4d3e-92ef-0c5f1e1b215a_isr_947248_26.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60f02b6ca185aa414a03b3f37d9908f9 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 5e025ed1bffea5938983458db9634c68 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6636480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/cbffd831-57d8-45e0-bfc7-54e885ac88bf_isr_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/cbffd831-57d8-45e0-bfc7-54e885ac88bf_isr_947246_49.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855ac2c2053b4fd31d742a38af48a15f with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash c80c0e49b3d780c98429de7a18ddcae3 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/cfba7de3-13f4-46e1-ab61-49f905a542a9_isr_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/cfba7de3-13f4-46e1-ab61-49f905a542a9_isr_947140_54.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adc618dc9028d5dda456491af49930c2 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 64b9ee8369cdbbb7295e4158a496d419 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/97bc108f-82c3-4c08-85f4-95f2c59abad3_characterizeImage_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/97bc108f-82c3-4c08-85f4-95f2c59abad3_characterizeImage_947140_4.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/abb413bd-6bd9-45a8-8c42-f76eadda4df5_isr_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/abb413bd-6bd9-45a8-8c42-f76eadda4df5_isr_947140_4.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc4073c3a928ac9538e60cbc749ab8e8 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash a35ad8c391b86b853392d32b8481aa10 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/3e8e8eb3-bd10-442e-a24a-79ad932445c5_characterizeImage_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/3e8e8eb3-bd10-442e-a24a-79ad932445c5_characterizeImage_947246_10.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1678de06-fae1-41a5-a51a-cda2391cac87_isr_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1678de06-fae1-41a5-a51a-cda2391cac87_isr_947246_10.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 023f3ae6b69d8904ca20834e097d844d with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 8363d7d28d21fb4af93edb6132f63ddf parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/d6b5b9e1-f256-4b2b-9ec1-79e3d0d6dae0_characterizeImage_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/d6b5b9e1-f256-4b2b-9ec1-79e3d0d6dae0_characterizeImage_947244_10.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f464b33f-fb6b-497a-b6c2-b4f83411d5b6_isr_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f464b33f-fb6b-497a-b6c2-b4f83411d5b6_isr_947244_10.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry deb44920fbc6101668d06d7448d78937 with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash df7dab5c8c4f635320218215cfcb0e51 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/932f2fe0-c54e-4826-adf9-463987f07737_characterizeImage_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/932f2fe0-c54e-4826-adf9-463987f07737_characterizeImage_947244_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7759db30-2bea-4d0f-9f93-d890e5340f3f_isr_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/7759db30-2bea-4d0f-9f93-d890e5340f3f_isr_947244_37.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc04bdc6655c61e39533088c6ec67156 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 832542642cc7c0684c170f0f05e398ed parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/193b5159-6112-450c-949d-a8a5c73f0a58_characterizeImage_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/193b5159-6112-450c-949d-a8a5c73f0a58_characterizeImage_947248_49.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/c0d3c63e-6c74-4e9f-b071-9303ac98bc8e_isr_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/c0d3c63e-6c74-4e9f-b071-9303ac98bc8e_isr_947248_49.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da45158d36dfc9011ad98ce4373b38fc with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 67ec77225c31fb32ea654d046a591ba1 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8be9c07f-f44a-48a4-a937-9ed5ffaedee0_characterizeImage_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8be9c07f-f44a-48a4-a937-9ed5ffaedee0_characterizeImage_947248_12.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/da6ad0ae-14f5-4a0d-9284-7c8f9cf7191b_isr_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/da6ad0ae-14f5-4a0d-9284-7c8f9cf7191b_isr_947248_12.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfefc5726e05cf7d41ba558ececb2055 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 18742ba57cf83ec3d6dea34ae29b7d31 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/70a6f99b-4139-447a-abce-55f291f7726b_characterizeImage_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/70a6f99b-4139-447a-abce-55f291f7726b_characterizeImage_947140_20.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/087fc3c3-a757-4181-a6c8-46e84767edb5_isr_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/087fc3c3-a757-4181-a6c8-46e84767edb5_isr_947140_20.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426c09d4968d07450bf38e6f2518305b with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 37da7a3bbc980109d2159e466fa87010 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d414efcd-7dc9-4174-8980-52150c97f168_characterizeImage_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d414efcd-7dc9-4174-8980-52150c97f168_characterizeImage_947144_40.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b823ca40-ba01-4d2f-a970-5d8ef43f4803_isr_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b823ca40-ba01-4d2f-a970-5d8ef43f4803_isr_947144_40.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 557fe07e29f97a5783dae79b63c3e188 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash cfa1ec056c8a4a72f9c062e339e8234d parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8ffb6ad5-3d42-450e-b824-8666af694bbf_characterizeImage_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8ffb6ad5-3d42-450e-b824-8666af694bbf_characterizeImage_947248_24.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/2ad16c68-2ce6-43ce-9361-2cec7ba44b30_isr_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/2ad16c68-2ce6-43ce-9361-2cec7ba44b30_isr_947248_24.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a1126f08051fc3ca5cfb97413e15be3 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 2174fd03cf1a6891de49159be1fddfab parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d57194a4-e535-4707-9575-c6a17c469d80_characterizeImage_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d57194a4-e535-4707-9575-c6a17c469d80_characterizeImage_947142_35.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f145a902-13cf-4d42-88de-38c30c7abf71_isr_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f145a902-13cf-4d42-88de-38c30c7abf71_isr_947142_35.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e09312fd2de239b29c9ef5552099702 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 0c9b8742cde94db522b458446ad60a01 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5214ea49-38f0-40bc-89db-32c95d28969a_characterizeImage_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5214ea49-38f0-40bc-89db-32c95d28969a_characterizeImage_947142_23.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e8789020-ebce-482b-9f96-1418c50b4034_isr_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e8789020-ebce-482b-9f96-1418c50b4034_isr_947142_23.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f62b72df57bf0b698af8cd67b843ae with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash b4b6e364ae736f78db86491c9c268cc7 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/241e08df-7a70-4a4b-9dd0-be207542dba3_characterizeImage_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/241e08df-7a70-4a4b-9dd0-be207542dba3_characterizeImage_947142_6.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/59457284-f812-4122-a814-b34e0cf6b0b2_isr_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/59457284-f812-4122-a814-b34e0cf6b0b2_isr_947142_6.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 821c6a768b6b825830d52d04c96f3f6b with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash afa3b30196885dcfcd455a392763a0b7 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3c68db00-2b4e-4879-ba2e-a32c1e060c1b_characterizeImage_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3c68db00-2b4e-4879-ba2e-a32c1e060c1b_characterizeImage_947144_41.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/144085b9-ed42-4c6b-afc4-4d1bc351c0a9_isr_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/144085b9-ed42-4c6b-afc4-4d1bc351c0a9_isr_947144_41.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216f0582b76cdf49fd58892edde542f1 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash ee7720abca20f442556dae4bb2fce668 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/163018e0-3865-4b8f-b779-49b25c9f8f59_characterizeImage_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/163018e0-3865-4b8f-b779-49b25c9f8f59_characterizeImage_947248_21.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7b333195-ba61-4bf7-a492-c02e7699cc3b_isr_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/7b333195-ba61-4bf7-a492-c02e7699cc3b_isr_947248_21.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ab4742e5c1c43b7334ba7334daedb9a with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash dde048d71d6b09e3db4030b095e2f488 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fc3a07a4-665e-41a2-beb1-f4a945699549_characterizeImage_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fc3a07a4-665e-41a2-beb1-f4a945699549_characterizeImage_947246_34.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1e89c651-1727-482e-ac41-98a7aa69c287_isr_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1e89c651-1727-482e-ac41-98a7aa69c287_isr_947246_34.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e5ddedd5e0fb6eb845cd76d32100e19 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash ec785c74eb132ee40046ba894b2b1a9c parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/913ff464-fbb5-4e9e-af51-e3905ef86e40_characterizeImage_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/913ff464-fbb5-4e9e-af51-e3905ef86e40_characterizeImage_947246_30.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ee64bfcd-bd59-4df5-948e-7ed2840ce1f6_isr_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ee64bfcd-bd59-4df5-948e-7ed2840ce1f6_isr_947246_30.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98fdd852787f95a639c78384bec186c1 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 7d7649eaff4af05cfb7b06c35bd1432a parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/559e970f-72d4-48d6-b8c2-c9db35286533_characterizeImage_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/559e970f-72d4-48d6-b8c2-c9db35286533_characterizeImage_947144_38.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/60d6a6e9-5548-4120-bf3b-9b508972d8c4_isr_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/60d6a6e9-5548-4120-bf3b-9b508972d8c4_isr_947144_38.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919802a1a0f20e84d420920bb8ed12ed with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 76b72ce5dc9478df07097f65b87c32c0 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7b8b129d-1e38-4baa-be13-4fb4c468349c_characterizeImage_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7b8b129d-1e38-4baa-be13-4fb4c468349c_characterizeImage_947248_38.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9b6e56d7-3e01-4640-b77c-97bf8b5c3f9c_isr_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9b6e56d7-3e01-4640-b77c-97bf8b5c3f9c_isr_947248_38.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f5d59801f95cd0f761f5c9dfc2e403 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 478b3856fe948d82a89cb5becaf79e9e parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ba2578c3-f50e-4889-bc7f-cb9c1df64067_characterizeImage_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ba2578c3-f50e-4889-bc7f-cb9c1df64067_characterizeImage_947244_47.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/688d89f2-9bd5-4f0b-9427-c16e75e52960_isr_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/688d89f2-9bd5-4f0b-9427-c16e75e52960_isr_947244_47.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47486b3f12e760b1909f19b5ca6995dc with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash e6d541015d6477319f526bb7ad29c9a3 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/e49cce65-8a12-4207-ab92-33918e518787_characterizeImage_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/e49cce65-8a12-4207-ab92-33918e518787_characterizeImage_947248_58.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/417043b5-5127-4ca8-b8f0-28f7ee913491_isr_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/417043b5-5127-4ca8-b8f0-28f7ee913491_isr_947248_58.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1dd9f6b23fae35ad66fb27fe507e99b with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 7fe37a00a40cddfd3a3d0b92cf191b78 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0cef5826-58fc-480d-a803-1cd82693d351_characterizeImage_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0cef5826-58fc-480d-a803-1cd82693d351_characterizeImage_947248_3.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/92634735-c8ba-48dc-9419-98513b8caadc_isr_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/92634735-c8ba-48dc-9419-98513b8caadc_isr_947248_3.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cda359439de8f18e8c06dffcb09538ca with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 5335be9eb348878bc7cf831886262e84 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:00 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e65b7015-0c73-423a-a14c-e5eac8c2d9ff_isr_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/e65b7015-0c73-423a-a14c-e5eac8c2d9ff_isr_947244_26.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b40778b09bcb4d156f7b475b24179eba with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 6f82d6542b14d4c5ea8f43f51454bceb parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d6adb8c5-edaa-493a-b014-809d6922e788_characterizeImage_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d6adb8c5-edaa-493a-b014-809d6922e788_characterizeImage_947142_8.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/92d4da0f-336c-450c-818a-370f68e0738b_isr_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/92d4da0f-336c-450c-818a-370f68e0738b_isr_947142_8.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c93ee0a1fe7d54fb450e8e9d8eb68c8 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 94d95da50032a68eaacddf2a7d52ad2e parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1cd1a098-e030-4136-88b1-d8263a65ee4f_characterizeImage_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1cd1a098-e030-4136-88b1-d8263a65ee4f_characterizeImage_947144_16.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/35e9ace6-4daf-48fe-bc42-b50d3ff06c07_isr_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/35e9ace6-4daf-48fe-bc42-b50d3ff06c07_isr_947144_16.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57fdc95f87e9d5e55792fbeb106d07c7 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 51fac95490741903c70908c93e94f0c7 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/07699eb7-a52d-48c5-92c3-9a5da11e0ba2_characterizeImage_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/07699eb7-a52d-48c5-92c3-9a5da11e0ba2_characterizeImage_947246_35.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2da539c8-8195-4282-ab98-d70929f36fe8_isr_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2da539c8-8195-4282-ab98-d70929f36fe8_isr_947246_35.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea69d418aea892a3b0f7329c396933c9 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash ab929729c1f5fd6392093a47e9c6d579 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0546d703-380e-498e-b4f6-505f45b0630f_characterizeImage_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0546d703-380e-498e-b4f6-505f45b0630f_characterizeImage_947248_59.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/81b8a874-4ef3-433f-8aeb-3387432541f0_isr_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/81b8a874-4ef3-433f-8aeb-3387432541f0_isr_947248_59.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07bc59f7668332c800448b48e1cb8477 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash af2591fd8fc58827a6c9fffc8a653fc6 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a18ae99b-1205-4485-a551-a594e7ba6cd4_characterizeImage_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a18ae99b-1205-4485-a551-a594e7ba6cd4_characterizeImage_947144_27.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7a0b755f-f9b0-4317-bb67-54e3c20d013e_isr_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7a0b755f-f9b0-4317-bb67-54e3c20d013e_isr_947144_27.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0af636d97ccc8ef97a26f61cae82139 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 1e55fe97fd85a6a5221bed37ccfa7d91 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d04ca1a2-eee2-48ae-a9dc-2d450624fd62_characterizeImage_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d04ca1a2-eee2-48ae-a9dc-2d450624fd62_characterizeImage_947246_39.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/edfff64e-6092-4c37-9fff-566f5257a848_isr_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/edfff64e-6092-4c37-9fff-566f5257a848_isr_947246_39.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 385a10382b0b227fcfb7b5de8dff56c7 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash d8a3623168894334eb3e187599afb223 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/23e5f658-d485-4e1e-bbc6-85205141d487_characterizeImage_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/23e5f658-d485-4e1e-bbc6-85205141d487_characterizeImage_947244_21.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4dce623b-264e-4bcd-861e-31d9053ae39c_isr_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4dce623b-264e-4bcd-861e-31d9053ae39c_isr_947244_21.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5544029ab6f31e2bc1a7303406e760c0 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash e257d7ac1b47da51ffd7f57ff9e216e0 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f0886063-00dc-4afa-ad65-cc9f7f22b608_characterizeImage_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f0886063-00dc-4afa-ad65-cc9f7f22b608_characterizeImage_947246_3.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/97c09209-1e20-4389-9268-a6f6df9545d7_isr_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/97c09209-1e20-4389-9268-a6f6df9545d7_isr_947246_3.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4a54c0d878d68b52e8b7f794f4bba3d with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 247e7560f1f011c3aba1b0f695838e83 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/72aaeccb-7c98-4a8b-8acd-0c18a1de95bc_isr_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/72aaeccb-7c98-4a8b-8acd-0c18a1de95bc_isr_947142_61.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6d9d76c6d6b143a54459e3cf67104b0 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 89cef9a5e13a6cfffd35e2fcb9e0814d parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/18f14bb5-3483-45dc-baa7-e04f1fe7b3a3_characterizeImage_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/18f14bb5-3483-45dc-baa7-e04f1fe7b3a3_characterizeImage_947144_11.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b513c30c-9ad9-43af-85f8-766fd1e7d4b0_isr_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/b513c30c-9ad9-43af-85f8-766fd1e7d4b0_isr_947144_11.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd20489fca73bbefc257bd1ca0dd715e with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 66e9b434eae81a8f4812f8dfd6a48c91 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e14f66a9-bf4a-41db-9993-385a321fa488_characterizeImage_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e14f66a9-bf4a-41db-9993-385a321fa488_characterizeImage_947140_7.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/322388a1-ad6f-407e-bcc0-0795c6c2de4f_isr_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/322388a1-ad6f-407e-bcc0-0795c6c2de4f_isr_947140_7.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee0eb6710837e725e9fbf08a21b9d17a with result from task 309 parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:01 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 52d74ef9d785a4d1fdf58e32f04bb90b parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bfe9667-a9fd-43d7-8fd5-56dd8fbf71f1_characterizeImage_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bfe9667-a9fd-43d7-8fd5-56dd8fbf71f1_characterizeImage_947142_24.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2dc57334-ce84-49db-98e5-be8d85fe835b_isr_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2dc57334-ce84-49db-98e5-be8d85fe835b_isr_947142_24.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae781dddba28c9d733326f37eb447efc with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash f0860680e0c00340471b73a6c844de2d parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b137cd3e-c8af-4608-875f-4918dd2c07a8_characterizeImage_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b137cd3e-c8af-4608-875f-4918dd2c07a8_characterizeImage_947142_9.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/866fa2e5-482d-4616-8a23-0f7578312b4c_isr_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/866fa2e5-482d-4616-8a23-0f7578312b4c_isr_947142_9.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4938be04d0f6269e836d8ff13263f96 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 263470a00f515e679ac23b3b6a19ae72 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0edd00a7-4415-436a-85ec-599eb719157e_characterizeImage_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0edd00a7-4415-436a-85ec-599eb719157e_characterizeImage_947248_43.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/11b89a65-4aa8-4eef-ac03-ce89f9d98c9b_isr_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/11b89a65-4aa8-4eef-ac03-ce89f9d98c9b_isr_947248_43.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac66915b978c456f60f2fb442d311811 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 2b8227d197b73e5248064cfe118482a3 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9949c787-83f8-4ed8-b0b3-53cf7fd19226_characterizeImage_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9949c787-83f8-4ed8-b0b3-53cf7fd19226_characterizeImage_947246_62.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/aa0a3bd7-d1ab-45cf-805c-477834424571_isr_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/aa0a3bd7-d1ab-45cf-805c-477834424571_isr_947246_62.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f7b9b1145e50bd428b2fad05447578c with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 0b17430d09f15ed6e725b421b8716727 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/682741b7-9b47-41e8-a06e-14656d01e9da_characterizeImage_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/682741b7-9b47-41e8-a06e-14656d01e9da_characterizeImage_947246_46.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/20680e9b-bc75-4529-9785-830bcc4352d5_isr_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/20680e9b-bc75-4529-9785-830bcc4352d5_isr_947246_46.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640091826e18a830b807cb0ef1415f9c with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash ffdde4c0e7cc88efec14383db4adddc4 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3e0f4186-de31-4905-ae37-f59ccfb9abdc_isr_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3e0f4186-de31-4905-ae37-f59ccfb9abdc_isr_947246_2.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4b6c5c7d345372d0482ead993fc46e0 with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash f66dccbdb745fd5472e8999bb2c88a8b parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f02a3f08-b30d-4649-816c-988d2656a830_characterizeImage_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f02a3f08-b30d-4649-816c-988d2656a830_characterizeImage_947246_58.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d804a5a9-11b6-4021-907c-41eb9470e2f2_isr_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d804a5a9-11b6-4021-907c-41eb9470e2f2_isr_947246_58.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bda8ddb3c2b3ff038a87b6a504769811 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 8429aa6485abf68827ea711eb9186e32 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/c0253d5a-03a5-4358-9012-9d91e90d64bd_characterizeImage_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/c0253d5a-03a5-4358-9012-9d91e90d64bd_characterizeImage_947142_56.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5bc1c319-bfd4-490b-b4c2-ca64a1f6ebae_isr_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5bc1c319-bfd4-490b-b4c2-ca64a1f6ebae_isr_947142_56.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0d2a760e35090103bc782853762be5 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 89d5662e1f021ba634a3f54ba662e942 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ccdf97ec-c26b-400b-8d38-8032de484f4d_characterizeImage_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ccdf97ec-c26b-400b-8d38-8032de484f4d_characterizeImage_947144_62.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0f1e8e4c-4589-42ae-ae70-d47bd5dfd6af_isr_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0f1e8e4c-4589-42ae-ae70-d47bd5dfd6af_isr_947144_62.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e121eb26954e3a3045646cea45b60e16 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 3722416e1ef352240b818e5ce970f967 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:01 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0fc41aca-6886-4dd0-9906-18b1d134ad22_characterizeImage_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0fc41aca-6886-4dd0-9906-18b1d134ad22_characterizeImage_947140_34.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b8bc19f0-8200-443e-90a5-be059f18cb76_isr_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b8bc19f0-8200-443e-90a5-be059f18cb76_isr_947140_34.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84752e95628c090a0148762bd7df375 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 481cd287556e6a58b8c5fad9b4217aca parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9773dc17-b175-4fbf-931f-bc0dca49fa09_characterizeImage_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9773dc17-b175-4fbf-931f-bc0dca49fa09_characterizeImage_947244_38.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/98b15f33-ea70-446d-9ece-e1fff1b2ba0a_isr_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/98b15f33-ea70-446d-9ece-e1fff1b2ba0a_isr_947244_38.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a588b92bd096e3913c2ca8c6702b282c with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash e6a33b7ba81e4d6691591ed047c6133e parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9d360107-31f8-4e44-868c-ed63af6ebc18_characterizeImage_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9d360107-31f8-4e44-868c-ed63af6ebc18_characterizeImage_947142_11.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e89f7c41-79cb-4e67-83f9-739df5e05a50_isr_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e89f7c41-79cb-4e67-83f9-739df5e05a50_isr_947142_11.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abfb88b39ee6182968b57a3c84bf972f with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 514193de9a982ad2bc5bd7fb42fc9297 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5496ad54-a1e3-45ce-b98a-b03d6b41bb0d_characterizeImage_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5496ad54-a1e3-45ce-b98a-b03d6b41bb0d_characterizeImage_947246_7.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3a8d595a-f6bc-4b6e-8d5c-4fb0bc33420e_isr_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3a8d595a-f6bc-4b6e-8d5c-4fb0bc33420e_isr_947246_7.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75e970a62dde9021d3ae41a69cf20990 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 881973277bdde2cb3c52d22f55f37e6a parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/52f890ac-bd2b-444b-b176-61e8250ddda2_characterizeImage_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/52f890ac-bd2b-444b-b176-61e8250ddda2_characterizeImage_947142_26.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/31f6fa17-00b4-421d-a291-cef089eaecbe_isr_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/31f6fa17-00b4-421d-a291-cef089eaecbe_isr_947142_26.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acb05ca39f687e981f118dbef6804782 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 46c749973ebde635fd49fc4b2676272c parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3968f56b-3311-4e58-972a-42a14058c4d8_characterizeImage_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3968f56b-3311-4e58-972a-42a14058c4d8_characterizeImage_947144_1.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/3a379e05-9764-41ed-a8d4-7ea65648de27_isr_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/3a379e05-9764-41ed-a8d4-7ea65648de27_isr_947144_1.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0143f443f73c09b15e5d746f9152f3 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 45fcea161a1cb4ed61a3e53686612444 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2fcbfb45-d388-40d4-8b75-8a361428e207_characterizeImage_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2fcbfb45-d388-40d4-8b75-8a361428e207_characterizeImage_947248_53.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/216ed1c8-ce38-4cdf-994b-301e4328cec3_isr_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/216ed1c8-ce38-4cdf-994b-301e4328cec3_isr_947248_53.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1fd0cf05d69636179aa65f9a0bb5ed7 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 3fabc6a8bd2e8b3b899250757c7957d8 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:01 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e9d05d97-4b4d-4758-b3ef-785ca963922e_characterizeImage_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e9d05d97-4b4d-4758-b3ef-785ca963922e_characterizeImage_947246_21.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/71f5032f-67da-4fc9-9647-77022894e589_isr_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/71f5032f-67da-4fc9-9647-77022894e589_isr_947246_21.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b09339841e91572281e286b120f17155 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 7f0e470a102f26c1706dc024d0906fad parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:02 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f5f8bdf8-9a28-4352-8c1e-3293441068bc_characterizeImage_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f5f8bdf8-9a28-4352-8c1e-3293441068bc_characterizeImage_947246_8.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0b19ea5f-6d6c-48d6-b0e5-4a9570fc6534_isr_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/0b19ea5f-6d6c-48d6-b0e5-4a9570fc6534_isr_947246_8.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34df103259d365df8442e4099ac2918b with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 06f7778b06966815189f23f29a33b7fb parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:03 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5cd9714c-395f-400b-975d-fa28832ca61f_characterizeImage_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5cd9714c-395f-400b-975d-fa28832ca61f_characterizeImage_947246_52.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ca5746c4-51bc-451a-bf83-6d823298eb3a_isr_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/ca5746c4-51bc-451a-bf83-6d823298eb3a_isr_947246_52.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db98b0065fc8fb07b955e1ae8c55239 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 6ee834de46fde9ab4716a48363f50028 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:05 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/56d108d5-fde0-4e0c-9cb4-a7f4ffcd07a0_characterizeImage_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/56d108d5-fde0-4e0c-9cb4-a7f4ffcd07a0_characterizeImage_947244_5.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b0e99148-1279-46c4-8ecd-bfed5d41edfb_isr_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b0e99148-1279-46c4-8ecd-bfed5d41edfb_isr_947244_5.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f094e6e6ade51215e862337e2e2d973 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 7d3e178f9b16066c14c8f8924608a921 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c25c04b4-47d6-40b0-ad96-217f9f0e0384_characterizeImage_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c25c04b4-47d6-40b0-ad96-217f9f0e0384_characterizeImage_947248_40.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/6bd83b7b-dbf7-494a-8c53-48cf7c480983_isr_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/6bd83b7b-dbf7-494a-8c53-48cf7c480983_isr_947248_40.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3980ec3f6bbbbd3fb2c1a1942c83ca6d with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 0c6e7e67c3bd80fd9d002b7037abcc8c parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/15fcc877-943f-4888-b5f1-619d1291cfb3_isr_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/15fcc877-943f-4888-b5f1-619d1291cfb3_isr_947144_17.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1459e3d5fb34844c099f35c6bc5c7675 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 81d0c67e33972fe2bec470382f84ff9f parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3e261ad8-1f21-47f4-9208-75f6eb888443_isr_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3e261ad8-1f21-47f4-9208-75f6eb888443_isr_947248_10.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d9b43f71c7b75c2f6936a9b7de980af with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash d5d50f699bfa41f62192eb0dd327e2ce parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fd0c7180-38a1-42a4-9685-a150136f08fb_characterizeImage_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fd0c7180-38a1-42a4-9685-a150136f08fb_characterizeImage_947246_5.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c93a326c-b3d4-49db-9f5c-e24e7a964d3b_isr_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c93a326c-b3d4-49db-9f5c-e24e7a964d3b_isr_947246_5.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249266b471bb429a1842d49f2ec33e7d with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash baf95c2cc6f155dd2c3be4f0eecb2397 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fc50836a-0bcd-47e2-b17c-f92fa7ff632d_characterizeImage_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fc50836a-0bcd-47e2-b17c-f92fa7ff632d_characterizeImage_947144_28.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f7995e2-e776-4046-a175-9f743d58d1fb_isr_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f7995e2-e776-4046-a175-9f743d58d1fb_isr_947144_28.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77a766288641926a19f0af41d9257055 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 8e94f9ccd8d08126e5810cb390e93f43 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/aa9be8d6-063c-4e83-87d8-3459594c2eab_characterizeImage_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/aa9be8d6-063c-4e83-87d8-3459594c2eab_characterizeImage_947144_39.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9a502fb9-1675-4981-ba0f-5d2518fca806_isr_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9a502fb9-1675-4981-ba0f-5d2518fca806_isr_947144_39.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 751e99801dcc2246d8e6b14436ae12fc with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 0ff48ff9b7e8eaede5af7bd902fdf21a parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7a77162e-f729-4e6e-bb02-ebf755b978a8_characterizeImage_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7a77162e-f729-4e6e-bb02-ebf755b978a8_characterizeImage_947144_50.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5178d66e-58fb-4738-ab0d-0e9d603fc26d_isr_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/5178d66e-58fb-4738-ab0d-0e9d603fc26d_isr_947144_50.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae280bf32f6e69f4a6b6411b0ade9c83 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash c673e76e9b928eb0c6cd5ddba7776b60 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:05 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/430b20cd-6033-4571-b530-236582da9c1a_characterizeImage_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/430b20cd-6033-4571-b530-236582da9c1a_characterizeImage_947244_45.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/ab9a82f6-2e0c-418f-884d-a3eeff625e9d_isr_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/ab9a82f6-2e0c-418f-884d-a3eeff625e9d_isr_947244_45.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24779355520b22842acb83df011faa7b with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 659e5c6424cc1d300f5517fab609dfde parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e71cd9f7-4c8a-4cca-9ee7-ec57492314f4_characterizeImage_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e71cd9f7-4c8a-4cca-9ee7-ec57492314f4_characterizeImage_947246_27.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/e8236567-a0bf-4177-b9ed-dc5c2b7e17c0_isr_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/e8236567-a0bf-4177-b9ed-dc5c2b7e17c0_isr_947246_27.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30fed63b09e1085e9dbf4cab9041b478 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 644918386555d176e43291846fbb91f2 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2247ca16-ba00-44c1-896c-d64d6ae6eeb7_characterizeImage_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2247ca16-ba00-44c1-896c-d64d6ae6eeb7_characterizeImage_947140_37.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/93f00889-092e-4447-871f-558d37d520e0_isr_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/93f00889-092e-4447-871f-558d37d520e0_isr_947140_37.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 408443df585f3d45eec8920b46119081 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 4d9c93bd066ea3398e566a49a119b8e2 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/1624d715-3ba0-4b45-8002-e4ed46d0da7b_characterizeImage_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/1624d715-3ba0-4b45-8002-e4ed46d0da7b_characterizeImage_947248_55.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/851e00c8-32a8-4a5d-80ba-db9f5c3b883d_isr_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/851e00c8-32a8-4a5d-80ba-db9f5c3b883d_isr_947248_55.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8005b05cc67c1bb3bc704869edbfb34d with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 33d5670453a0e23dc9bba0db1a4a63d7 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d10b97d9-3010-433a-9df5-b708991e751c_characterizeImage_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d10b97d9-3010-433a-9df5-b708991e751c_characterizeImage_947246_41.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b066496b-4206-4e24-801e-56b5d4ae61db_isr_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b066496b-4206-4e24-801e-56b5d4ae61db_isr_947246_41.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2829117f23b6ba40127785f6143807 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 11bd99419a955c2b1e71dea3c6c7476a parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/6c58e594-543c-4f5d-a8da-31eea6211585_characterizeImage_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/6c58e594-543c-4f5d-a8da-31eea6211585_characterizeImage_947246_40.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/af15b740-49d4-421d-91af-a3abc9957a34_isr_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/af15b740-49d4-421d-91af-a3abc9957a34_isr_947246_40.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 602c0b729b971a88c2676fb356b6ef4c with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 25789eb1f24029ec96fd19f6111ac421 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/27023f8d-fa91-4998-9037-c41cee00af06_characterizeImage_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/27023f8d-fa91-4998-9037-c41cee00af06_characterizeImage_947248_8.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9a113520-2d4e-4d63-9bd2-4b35f8417010_isr_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9a113520-2d4e-4d63-9bd2-4b35f8417010_isr_947248_8.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32bb767b83d0197fb3aedf9a029ac443 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 7e7eadaa9017b7dfe3b71493d77adfad parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a706c7c4-b726-458d-98ec-4bef47b60ed5_characterizeImage_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a706c7c4-b726-458d-98ec-4bef47b60ed5_characterizeImage_947246_19.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d6cd5981-20b1-4618-9421-c63e233202f4_isr_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d6cd5981-20b1-4618-9421-c63e233202f4_isr_947246_19.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3185d1a952051ad932561327e1d0758a with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 33ad16bba09b4abe4db9909babc1efc6 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/bd5edb29-6ac5-49b7-9944-40794d421a2e_characterizeImage_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/bd5edb29-6ac5-49b7-9944-40794d421a2e_characterizeImage_947140_33.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c5792c44-8330-426e-acd1-6bbf349d4038_isr_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c5792c44-8330-426e-acd1-6bbf349d4038_isr_947140_33.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea548e55b066637ca96b3b70e8bfe5a with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 764e347edfb55f3690428ed720425b55 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/582c43cb-357a-4b08-8fe4-8d53a5be365d_characterizeImage_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/582c43cb-357a-4b08-8fe4-8d53a5be365d_characterizeImage_947244_39.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8ffc0d7f-bb15-4891-99f9-e0f1c9aaa795_isr_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8ffc0d7f-bb15-4891-99f9-e0f1c9aaa795_isr_947244_39.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f99e4590a3fb2d854a2a7f30ea726515 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 8cf015058304821bce5eb6901a56319f parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/25edbba0-a670-47c5-a7a3-4e9a3940d7c4_characterizeImage_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/25edbba0-a670-47c5-a7a3-4e9a3940d7c4_characterizeImage_947144_19.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0abd8e38-4f4d-4722-ba73-7a974157628c_isr_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0abd8e38-4f4d-4722-ba73-7a974157628c_isr_947144_19.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfc7dddaec82a1b3211dbb8eaf40260d with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 6537c95e96476300deebfbb8b7b9fd1d parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c64f2059-9c1c-4cf0-82e5-50d58b36f72f_characterizeImage_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c64f2059-9c1c-4cf0-82e5-50d58b36f72f_characterizeImage_947246_13.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/480d881f-fdd9-4324-b6da-c4d55ed0625d_isr_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/480d881f-fdd9-4324-b6da-c4d55ed0625d_isr_947246_13.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f66892cb736008aa6747b72c5e2c9eac with result from task 474 parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:06 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 2ac5b0569b33b3d76e54c6524bb25e65 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/01b32ded-fc56-409b-ac32-b83184ca5db6_characterizeImage_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/01b32ded-fc56-409b-ac32-b83184ca5db6_characterizeImage_947140_53.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e926dadc-1225-43a9-a288-f3ce31ea360f_isr_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e926dadc-1225-43a9-a288-f3ce31ea360f_isr_947140_53.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3c7bbbf3883a963860463cfdc67a9f with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 477780e0bb56ea66eadd2162021152ea parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:06 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d2c3c97-7fc1-4eae-9524-8952b7aac9f8_characterizeImage_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d2c3c97-7fc1-4eae-9524-8952b7aac9f8_characterizeImage_947140_31.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/700ceabc-66d3-40bb-a9c6-46b1f31969fe_isr_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/700ceabc-66d3-40bb-a9c6-46b1f31969fe_isr_947140_31.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e284fe1dcd26804ffa56b081a109149 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 8f012394d71a966f59d3deadc23b5614 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031604 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:07 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031605 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:26 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/50fe245e-f669-433c-91ed-87bd304916d6_characterizeImage_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/50fe245e-f669-433c-91ed-87bd304916d6_characterizeImage_947248_25.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d3c9794a-1901-4903-be0b-9794dd5a7ac2_isr_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d3c9794a-1901-4903-be0b-9794dd5a7ac2_isr_947248_25.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8083c88ef9766c24c5fc78d19eaed11c with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 14c679ff8e9af33e3b24391e8d64bb55 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:56 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0d0ccf5c-2d78-4e5b-a83d-50a2f12b09be_characterizeImage_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0d0ccf5c-2d78-4e5b-a83d-50a2f12b09be_characterizeImage_947246_51.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c0f465c9-aac5-4450-986d-987753fd8855_isr_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c0f465c9-aac5-4450-986d-987753fd8855_isr_947246_51.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 116013d54f7da253d9aa1cbd008fd139 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 667291a4d330d2c9e21660dc4fbb8d8c parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d7e82b5e-222a-43a5-83ab-116c7d9e0f73_characterizeImage_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d7e82b5e-222a-43a5-83ab-116c7d9e0f73_characterizeImage_947140_60.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/15f7d346-263e-4443-b06c-f2098c5788f5_isr_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/15f7d346-263e-4443-b06c-f2098c5788f5_isr_947140_60.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb1022736caeb140bd6bd84ac8881bc4 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 60c45864565b28a2e258bf04f0a5f4a1 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bf652a6-6df6-4269-9ab6-13d597dd0c41_characterizeImage_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bf652a6-6df6-4269-9ab6-13d597dd0c41_characterizeImage_947142_1.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c8fa4a7c-0f6e-42da-84f9-2784cf56ec23_isr_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c8fa4a7c-0f6e-42da-84f9-2784cf56ec23_isr_947142_1.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb2fe7567bda78be60d2ff077c60e14a with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash c6dcf18f8c292581a31cd011f68882c0 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/3a39456f-e044-4868-a4d5-d524ffab480f_characterizeImage_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/3a39456f-e044-4868-a4d5-d524ffab480f_characterizeImage_947140_41.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f60cd11-0ba2-4658-8dc7-26c1d127d041_isr_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f60cd11-0ba2-4658-8dc7-26c1d127d041_isr_947140_41.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19252940fccfee18b1489d1c82c1266 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash ff0e6b11bb6a82043aca36077b275a96 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/fdeb31eb-c3b6-4cd0-846a-17f602f50194_characterizeImage_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/fdeb31eb-c3b6-4cd0-846a-17f602f50194_characterizeImage_947140_1.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/943f8244-48a6-4ad6-9661-bc0dc6989c48_isr_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/943f8244-48a6-4ad6-9661-bc0dc6989c48_isr_947140_1.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0889add53e0498cd1d85e2e6c18ee13d with result from task 411 parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash e6e265f4a6f40d6f0f2150b92a37faa6 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:57 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e80e5349-99c0-43c6-a43d-61a83ea423ee_characterizeImage_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e80e5349-99c0-43c6-a43d-61a83ea423ee_characterizeImage_947144_29.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/43fdbf49-cbdd-4b08-8865-dd43c73b8ad2_isr_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/43fdbf49-cbdd-4b08-8865-dd43c73b8ad2_isr_947144_29.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6674ec5433e9da446e00fba8cd128dcb with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash ee87bb2ff6440f47d4942984c58e27db parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0b5f3abb-1e31-4f85-91ee-46182a7a79b8_characterizeImage_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0b5f3abb-1e31-4f85-91ee-46182a7a79b8_characterizeImage_947244_22.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fdbafef9-f7c0-4ea4-878b-48d341e325df_isr_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/fdbafef9-f7c0-4ea4-878b-48d341e325df_isr_947244_22.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 585b025edbd0c9242577bcbd4d205cc1 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash a224d0962d8ab618b7b17b07f96d2096 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5ad1e235-91ba-4db6-ae4b-f577ed9a535d_characterizeImage_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5ad1e235-91ba-4db6-ae4b-f577ed9a535d_characterizeImage_947244_49.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cb903a-d03e-45fc-9830-b6ac54686b90_isr_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cb903a-d03e-45fc-9830-b6ac54686b90_isr_947244_49.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 921099fa068ddb27827335a7735b6587 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash b7078d1a9763e5900bbcefee41570612 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/07f3bdb1-90b3-465e-9f6c-ed569b1360de_characterizeImage_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/07f3bdb1-90b3-465e-9f6c-ed569b1360de_characterizeImage_947244_28.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/49f16a2a-8e7d-493c-8ede-0a70450e5198_isr_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/49f16a2a-8e7d-493c-8ede-0a70450e5198_isr_947244_28.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9f64395163897d2672bf4e877eedaf with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 118f63f541a451c9928b45e090b8f0ee parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6fe6b9c6-18ee-416c-88a1-dac3e4305ddd_characterizeImage_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6fe6b9c6-18ee-416c-88a1-dac3e4305ddd_characterizeImage_947144_4.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/4481e27f-f58b-4982-9811-f8ced0095776_isr_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/4481e27f-f58b-4982-9811-f8ced0095776_isr_947144_4.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e0ba9a3d3b7287744a1b75851e0e66 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 143af9b46a2249118eb254c8f19c7962 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8503d3c4-2094-4389-8761-fd3eabe3757a_characterizeImage_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8503d3c4-2094-4389-8761-fd3eabe3757a_characterizeImage_947140_28.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/1e244e3e-5f12-4956-9332-9b6283b378e9_isr_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/1e244e3e-5f12-4956-9332-9b6283b378e9_isr_947140_28.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 842273b9de08e1a8c5caada7e95f4964 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 0e9cd3ddb4a6db8be0b8cf82f1e0878f parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:57 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f147fef3-43e8-4a57-808a-d7706b06bf5a_characterizeImage_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f147fef3-43e8-4a57-808a-d7706b06bf5a_characterizeImage_947248_19.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/0ff264af-76c6-4a48-917b-e39a9e23e0c7_isr_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/0ff264af-76c6-4a48-917b-e39a9e23e0c7_isr_947248_19.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4974dc1c131066a9d42a9d1b21825ff with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 1c6277763320ef3ac3b2f72960c1b901 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:35:59 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/6c38d1d7-a977-4531-b019-9f9408b742df_characterizeImage_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/6c38d1d7-a977-4531-b019-9f9408b742df_characterizeImage_947244_7.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45b5b37-71ed-4f99-9b04-23bcde05b6fa_isr_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45b5b37-71ed-4f99-9b04-23bcde05b6fa_isr_947244_7.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2ca00914e65e494789b7153b8af70de with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 6417472c9dfbce20848e974c343e4904 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2339d140-3397-413a-a288-52ca9a8c1e59_characterizeImage_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2339d140-3397-413a-a288-52ca9a8c1e59_characterizeImage_947142_60.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fbaa49c0-dc80-421a-bbc3-8af3ada00b21_isr_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fbaa49c0-dc80-421a-bbc3-8af3ada00b21_isr_947142_60.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d11f7eb82e82fdabfc76ef805ffda72 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 17108cb988fa9bfbc8816f60b7972700 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:00 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:01 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/825af127-f636-406e-9b67-73e8cbb9aa19_isr_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/825af127-f636-406e-9b67-73e8cbb9aa19_isr_947140_61.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b93e65287295b2b73f2f4cb7f3954715 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash fa37f494feecfc5a596ba39456a1e6ba parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/401f1426-b0c3-4e83-9784-e6f44e5560bc_characterizeImage_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/401f1426-b0c3-4e83-9784-e6f44e5560bc_characterizeImage_947142_44.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c4d313f8-de5c-46c6-9e2c-0450fadb2563_isr_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c4d313f8-de5c-46c6-9e2c-0450fadb2563_isr_947142_44.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584f89fc5707f558c9e7bd25b1ff9b5c with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 2f20f22b8cb8ddf62124354a26c633d5 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/a6e93949-2718-4b9f-a300-cd41fc1da6b6_characterizeImage_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/a6e93949-2718-4b9f-a300-cd41fc1da6b6_characterizeImage_947244_17.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/caf74f01-24e7-497d-b884-1356f61557aa_isr_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/caf74f01-24e7-497d-b884-1356f61557aa_isr_947244_17.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7137d8bebc8798266b5ed5cd0b482717 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash e0106c5ba4f7390651b0f7067724aef3 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:01 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/00da1f83-3245-490d-81d5-c6dbbd0cf03a_characterizeImage_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/00da1f83-3245-490d-81d5-c6dbbd0cf03a_characterizeImage_947244_33.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3f2c16c2-08a1-4108-97e0-6e9cfc956372_isr_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3f2c16c2-08a1-4108-97e0-6e9cfc956372_isr_947244_33.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d553f6f9a39e8c0594979ca03097a7fb with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash abd107ac91dd244ef5908a8f2c7c33b2 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/18f1382d-1e08-40e8-bab2-864d0b8eea74_characterizeImage_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/18f1382d-1e08-40e8-bab2-864d0b8eea74_characterizeImage_947248_23.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1808a4cb-a248-4c3e-869a-70fb3db951e1_isr_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/1808a4cb-a248-4c3e-869a-70fb3db951e1_isr_947248_23.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5c213979972ee18ef2d5a15531af820 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 064da76330d1cf53ee282b56a13e3b07 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:03 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:04 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/bf64f706-93a4-4e93-a4d1-193c46a9553e_characterizeImage_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/bf64f706-93a4-4e93-a4d1-193c46a9553e_characterizeImage_947246_50.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/5d99e4ae-ccfc-403e-b225-464c4484b1ce_isr_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/5d99e4ae-ccfc-403e-b225-464c4484b1ce_isr_947246_50.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86909df2601170c49a43d3d197e0049f with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 1eca1e3035417e7300905b93079f8e52 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2c189336-1fdb-447f-b32a-58d8b6b5b3cb_characterizeImage_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2c189336-1fdb-447f-b32a-58d8b6b5b3cb_characterizeImage_947246_53.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/6ba62a55-3929-4ce3-96a9-927da04e1c94_isr_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/6ba62a55-3929-4ce3-96a9-927da04e1c94_isr_947246_53.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72070668ffe61ff9d92d07b59fe57e89 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 876b5f124cd82df98aaaf2742bca3da8 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/330ba905-3755-4fcf-ac64-ca150896e7d8_characterizeImage_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/330ba905-3755-4fcf-ac64-ca150896e7d8_characterizeImage_947248_31.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a8b795d8-3d6c-4bc0-8028-eef98309b591_isr_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a8b795d8-3d6c-4bc0-8028-eef98309b591_isr_947248_31.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7444b7e4059d5326e90c47eea85b10ba with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash a3d5d376fd9d42c700a8f5f4c5eb612d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6b540640-a0a7-4a71-95aa-dbbc4cc0e4a2_characterizeImage_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6b540640-a0a7-4a71-95aa-dbbc4cc0e4a2_characterizeImage_947144_34.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d9676206-3965-494c-8977-c00d545dc129_isr_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d9676206-3965-494c-8977-c00d545dc129_isr_947144_34.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b042258687b06be9c347bfc6a4d04f6 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 1d780c87d9ef0b0446a6e04f5bc37ad9 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:07 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/dbd761ce-b293-4a82-8977-752bd018fab8_characterizeImage_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/dbd761ce-b293-4a82-8977-752bd018fab8_characterizeImage_947142_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3167c557-e9fa-4ab9-a144-1c744dfd59a2_isr_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3167c557-e9fa-4ab9-a144-1c744dfd59a2_isr_947142_32.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 727669731e586986e3da4a3afcde6ede with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 4f40914c65f09e05209777f30898599d parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/da69d1b2-9c82-4ec5-9bbc-e60e71df2707_characterizeImage_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/da69d1b2-9c82-4ec5-9bbc-e60e71df2707_characterizeImage_947142_39.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a285015f-3a50-43a8-a82f-4eb86819d0fa_isr_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/a285015f-3a50-43a8-a82f-4eb86819d0fa_isr_947142_39.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ba01e51065950148e8debb2d7162c4 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 894a3ccdadcb7f19dfe96878fc538745 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:09 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/87294cf0-5207-4d53-95bb-98f7c9492212_characterizeImage_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/87294cf0-5207-4d53-95bb-98f7c9492212_characterizeImage_947246_57.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c5e8e0db-7cc4-40c3-b771-244bfc7ce810_isr_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c5e8e0db-7cc4-40c3-b771-244bfc7ce810_isr_947246_57.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a08d122851e62203078d00491ed8d5cd with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:11 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 03d74547b3bffa077458d60f708a42a9 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/950ea26a-1ed4-447b-8a90-b6019137f296_characterizeImage_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/950ea26a-1ed4-447b-8a90-b6019137f296_characterizeImage_947142_42.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/82702a4f-6413-436d-aa99-034f135f1abc_isr_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/82702a4f-6413-436d-aa99-034f135f1abc_isr_947142_42.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2513e16dbeb54c2663b3d5af9cf0189 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 898331e8f1cc42bd323dc6ffa57a6316 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:11 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:12 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f778265d-805d-4589-9d25-a114c256b209_characterizeImage_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f778265d-805d-4589-9d25-a114c256b209_characterizeImage_947144_10.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f0241327-9c15-4637-a75e-cc31f86d3c40_isr_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f0241327-9c15-4637-a75e-cc31f86d3c40_isr_947144_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 0 failed parsl.dataflow.dflow INFO: Task 214 marked for retry parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash bf99c1e847d3fccc5fb91e6f18040f61 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 1 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4bcb2bafaa0dc6af07d87e87549c9f4 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash b7eec916d3cf0b2a6f67dff9c39f657a parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/108b697b-ecd0-472d-a44f-a8d7ba4a5798_characterizeImage_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/108b697b-ecd0-472d-a44f-a8d7ba4a5798_characterizeImage_947244_50.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/76819225-79be-453d-b6cb-232fabafb428_isr_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/76819225-79be-453d-b6cb-232fabafb428_isr_947244_50.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3a5e748744aba019ef121e8fd8ea7ca with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash adb9439c2a5a6d46c534fb33f54ead33 parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a77b302e-bd62-4ec3-bb57-b9bcb691e54b_characterizeImage_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a77b302e-bd62-4ec3-bb57-b9bcb691e54b_characterizeImage_947142_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c7025827-aea3-4a4d-bcdc-4e4928fc3cf1_isr_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/c7025827-aea3-4a4d-bcdc-4e4928fc3cf1_isr_947142_21.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7ce7c1de5fa66169f6a1f03ae5e1f8 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash fd1840cf99dffaed63d627df6ed9d802 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/531db1c2-6f99-4cd3-998e-f9b9eec9d899_characterizeImage_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/531db1c2-6f99-4cd3-998e-f9b9eec9d899_characterizeImage_947140_17.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3db45e4a-83db-4d52-9ec4-8a1f10ffe3b6_isr_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3db45e4a-83db-4d52-9ec4-8a1f10ffe3b6_isr_947140_17.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c96f3e3d7a5ff4d79efe1af18ae5bc1 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 02c99881b176188437d4a993367f2e38 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a934af7e-f0e1-4037-ada1-1419e041902d_characterizeImage_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a934af7e-f0e1-4037-ada1-1419e041902d_characterizeImage_947142_54.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b969e660-cb72-49d5-b4d2-e9a99ab0b774_isr_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/b969e660-cb72-49d5-b4d2-e9a99ab0b774_isr_947142_54.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6641748591c9e1672d859d0e17423353 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash ec9a57a3416e829e1ac7b67f4b697171 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/783d843a-48fc-4268-9ba9-748730a9f1e2_characterizeImage_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/783d843a-48fc-4268-9ba9-748730a9f1e2_characterizeImage_947142_43.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e4493787-4b40-4a5a-841a-278baa6eb347_isr_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/e4493787-4b40-4a5a-841a-278baa6eb347_isr_947142_43.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7bb9dc31f008e51f01d9083fd9f1e64 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 336fb8a33b5d4bb6e98860627b87b4cc parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ed641de2-1f68-44e8-8268-738bef07aa4a_characterizeImage_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ed641de2-1f68-44e8-8268-738bef07aa4a_characterizeImage_947144_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/99bc3611-cad1-4d02-b1db-256d59e89738_isr_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/99bc3611-cad1-4d02-b1db-256d59e89738_isr_947144_20.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6958d5eb68068b7d312f77cd26af1811 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 43cb8537feaf95bb479a4db3bc0d8b26 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b611a4ce-6684-413b-8e0c-f9f9033f36f1_characterizeImage_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b611a4ce-6684-413b-8e0c-f9f9033f36f1_characterizeImage_947142_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/57b9bae6-526e-40a0-bd8e-10122370da5a_isr_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/57b9bae6-526e-40a0-bd8e-10122370da5a_isr_947142_38.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6772a4a91d011d7e2da3ee7f86d8977 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 0369a41fee84cfcf726cd8054d4e8d6c parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/280c5f24-4da2-41ff-a401-ba1ceded3630_characterizeImage_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/280c5f24-4da2-41ff-a401-ba1ceded3630_characterizeImage_947144_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/78a7c94e-9303-4bce-bf11-4b4b75170103_isr_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/78a7c94e-9303-4bce-bf11-4b4b75170103_isr_947144_47.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04121112e46194a5fd3a72f315ba85cc with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 16a6bdde378d2418f3aaa67930f970da parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4e3fcf5d-7986-49e8-9a88-d837582bded7_characterizeImage_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4e3fcf5d-7986-49e8-9a88-d837582bded7_characterizeImage_947248_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cce60aa9-f827-47c6-a290-efa574d8612a_isr_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cce60aa9-f827-47c6-a290-efa574d8612a_isr_947248_47.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29850869d47375a155c1f7defff9db0d with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 880de859be19b8f9378c661e6d2fbfc5 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a7a3c4f8-9f5c-4d91-b934-f439cd1f17d8_characterizeImage_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a7a3c4f8-9f5c-4d91-b934-f439cd1f17d8_characterizeImage_947246_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c696e03e-b245-4ac1-a8f9-deac03348146_isr_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c696e03e-b245-4ac1-a8f9-deac03348146_isr_947246_1.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54700ba47a6aa89d05fe047703a988aa with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 0a45c14e655a253635d663241a8551fb parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e56fafc7-4755-457b-b3cb-16b521023f69_characterizeImage_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e56fafc7-4755-457b-b3cb-16b521023f69_characterizeImage_947144_57.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f6c970c9-682a-4d16-9c7b-22923d213cec_isr_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/f6c970c9-682a-4d16-9c7b-22923d213cec_isr_947144_57.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91a3a16785272eda7aecacc905588e0 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash c47bd21e6306e87b475dba117c338e17 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:14 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/69a7f9be-e877-4b96-8ffc-efd67708d9db_characterizeImage_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/69a7f9be-e877-4b96-8ffc-efd67708d9db_characterizeImage_947248_20.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/86e37b70-db96-449d-8b37-f3e1461050e6_isr_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/86e37b70-db96-449d-8b37-f3e1461050e6_isr_947248_20.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb8c163a3bc0c4ae5fd5803d5c22a86 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 23a030a467260a213a5d82a9fdbdf500 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bda16147-9694-40f2-98a7-ef23850f42f6_characterizeImage_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bda16147-9694-40f2-98a7-ef23850f42f6_characterizeImage_947248_13.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/49c1d9fe-210c-478a-8a54-83c10463c986_isr_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/49c1d9fe-210c-478a-8a54-83c10463c986_isr_947248_13.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31140d3b03eb2e796a140a62d33409bc with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 534e6629004402dc2d4ce9c58e5c7173 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d5e8eb69-5ea2-4c60-b60e-300f8aafe6fa_characterizeImage_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d5e8eb69-5ea2-4c60-b60e-300f8aafe6fa_characterizeImage_947140_14.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b09b5b14-4fdc-4126-abc6-29902f965c94_isr_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b09b5b14-4fdc-4126-abc6-29902f965c94_isr_947140_14.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c171482327fbc78b64b98dab928d39 with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 580442d1fb0aa95fd2e78a735555c99d parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/ea73a511-331d-4d26-b44e-9b4b7ca44fe4_characterizeImage_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/ea73a511-331d-4d26-b44e-9b4b7ca44fe4_characterizeImage_947142_51.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5ca77c97-9608-4b25-a308-c5103b7e9d6f_isr_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/5ca77c97-9608-4b25-a308-c5103b7e9d6f_isr_947142_51.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749d43316d78ccb59052b25b223e4888 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 8587dda83241ed5db074085b5172d482 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0bfd9cf5-61e5-47ca-a9cf-bbc7518da789_characterizeImage_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0bfd9cf5-61e5-47ca-a9cf-bbc7518da789_characterizeImage_947246_18.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2911c989-713e-427f-bf71-6f4c220eea71_isr_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/2911c989-713e-427f-bf71-6f4c220eea71_isr_947246_18.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baac75837c7f21e412bc560be645291d with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 7e5a8d25ce260e578152a39e07bddf52 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:14 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3214bbbd-21b8-40ca-9416-bfd77162ddf8_characterizeImage_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3214bbbd-21b8-40ca-9416-bfd77162ddf8_characterizeImage_947248_6.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/dbe88134-dc97-4088-97fc-927cc9b9422d_isr_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/dbe88134-dc97-4088-97fc-927cc9b9422d_isr_947248_6.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 814155bc49f1f1c883dcdf63df297324 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash fbe42cd72aed92e4df6d7e1d52c1bc93 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/2e085133-842c-4701-866c-0ce12dca1e30_characterizeImage_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/2e085133-842c-4701-866c-0ce12dca1e30_characterizeImage_947144_13.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e41a827f-98b5-4fa1-94ae-d076b9f1d167_isr_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/e41a827f-98b5-4fa1-94ae-d076b9f1d167_isr_947144_13.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aca6265ccbe032b7884ee14c52e3329 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 4e7092410d6cad15601bfbcf4716342c parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/42fdefcb-c66e-4358-ae4f-537f1cae7bb6_characterizeImage_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/42fdefcb-c66e-4358-ae4f-537f1cae7bb6_characterizeImage_947246_38.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b6a14c86-7052-44ba-864c-7b9fad19f07a_isr_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b6a14c86-7052-44ba-864c-7b9fad19f07a_isr_947246_38.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6e1c6bba2d415214a57887c1a9b0aeb with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash a975e8aae63573627d549dc7ba15ef1f parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d34b0541-7afc-48b2-a133-a358165ea376_characterizeImage_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d34b0541-7afc-48b2-a133-a358165ea376_characterizeImage_947246_32.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/24d0cfc5-8a91-442f-ba64-4167ddccd516_isr_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/24d0cfc5-8a91-442f-ba64-4167ddccd516_isr_947246_32.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8e78aa67ba51b97363eb98687db208 with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash e200fd3f81f4378f66dede93ad9086eb parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d689a836-6193-4ed8-8203-7ba42874e78b_characterizeImage_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d689a836-6193-4ed8-8203-7ba42874e78b_characterizeImage_947142_17.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/cf61768d-0899-48c8-b8f7-ebe0940077f0_isr_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/cf61768d-0899-48c8-b8f7-ebe0940077f0_isr_947142_17.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3578a51e200cd6942c427553913aa70 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash b7d095e09d00e3239f791598dc3379a5 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e21cdfe1-4e0d-4ffd-8160-4641094fd3af_characterizeImage_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e21cdfe1-4e0d-4ffd-8160-4641094fd3af_characterizeImage_947244_25.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/355cfcc2-3519-4434-904a-8780db3c314a_isr_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/355cfcc2-3519-4434-904a-8780db3c314a_isr_947244_25.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 160de91fbfa884093049af4c999f8df2 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 68999cb9f7baa360a4a500f8f302ed63 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0bab7182-0491-4aff-826f-d1446addde01_characterizeImage_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0bab7182-0491-4aff-826f-d1446addde01_characterizeImage_947248_22.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fdbf973e-3f55-4571-b931-2ad8e24b4f0d_isr_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fdbf973e-3f55-4571-b931-2ad8e24b4f0d_isr_947248_22.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcc9ae316526e1115b5e135df0a3e98b with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash e65af1337e8b9f1ae48fac22f216d578 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fe38f9f9-5674-4dc9-860a-8f19a96bde89_characterizeImage_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fe38f9f9-5674-4dc9-860a-8f19a96bde89_characterizeImage_947144_12.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/fc439d27-c2d6-4497-8cdb-ac8cf20d7de7_isr_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/fc439d27-c2d6-4497-8cdb-ac8cf20d7de7_isr_947144_12.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8d9654b8037a1c99b3ff19809c46ad7 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash fc9c35abf187bd13e1e79209c965a8dc parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/943d926b-22da-41b1-8bd3-7d4ec728bd88_characterizeImage_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/943d926b-22da-41b1-8bd3-7d4ec728bd88_characterizeImage_947248_27.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/619c025c-8d10-449e-828d-107b46cfadf7_isr_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/619c025c-8d10-449e-828d-107b46cfadf7_isr_947248_27.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fce09923053fc081e7910c7a1bd7606a with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash c9ab4c4ddd0cd5ff9e2e33d5fc726a0a parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 CG 17031604 CG 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 126 parsl.dataflow.strategy DEBUG: Executor multi has 126 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:36:17 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c68208e-537a-4702-8800-3e371782a235_isr_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4c68208e-537a-4702-8800-3e371782a235_isr_947142_2.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa6459b74128a4d5f59c3d270518adfd with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash e406c12c5482dc78124b6d3ebcdfedd7 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 05:36:17 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e74e3349-954d-4241-ab2d-303f4231cfce_characterizeImage_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e74e3349-954d-4241-ab2d-303f4231cfce_characterizeImage_947244_14.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6aa115ea-5eed-411a-98b4-1acc2920b18a_isr_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6aa115ea-5eed-411a-98b4-1acc2920b18a_isr_947244_14.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2323639679d6efe597c1703a1fb8abb with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 7f5a32bf16e801953fc30f9166f1a046 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a6db2017-3019-4db6-9217-87ca388b9823_characterizeImage_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a6db2017-3019-4db6-9217-87ca388b9823_characterizeImage_947144_60.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7bb7cfc6-b326-4a3c-afe2-a6f4d5c47063_isr_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7bb7cfc6-b326-4a3c-afe2-a6f4d5c47063_isr_947144_60.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c20fe70a5f8adfc152f465de2bdbcddc with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 950f10bffef14d45ae27cf2ba92b3bf9 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:19 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:22 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/35a4c333-2047-4bb3-be3e-b8c79f959147_characterizeImage_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/35a4c333-2047-4bb3-be3e-b8c79f959147_characterizeImage_947248_28.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a5ce70a9-8c71-482f-b3a7-1593805db8ea_isr_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/a5ce70a9-8c71-482f-b3a7-1593805db8ea_isr_947248_28.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 553eb671483baad2d4f6a239764eec2f with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 2d11463ab5d19ca7acfdcb39f61dc4c6 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:23 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/b4a62b0e-9c30-4efc-b0e5-7d630d0731a0_characterizeImage_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/b4a62b0e-9c30-4efc-b0e5-7d630d0731a0_characterizeImage_947140_18.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f237b5a8-0cbd-41ff-9587-596b2eaab252_isr_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f237b5a8-0cbd-41ff-9587-596b2eaab252_isr_947140_18.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6a1c4b90ff0072c75d5216255347930 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 2591d132eb169153d22efb2f3c5c3267 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:24 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d14292a0-106d-45fd-a0e3-153d96cca31f_characterizeImage_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d14292a0-106d-45fd-a0e3-153d96cca31f_characterizeImage_947144_44.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/bf529ea2-2e7a-4b43-b3d6-36c8fe6b2856_isr_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/bf529ea2-2e7a-4b43-b3d6-36c8fe6b2856_isr_947144_44.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b57848ace7e8d05cb15c667fe33b544 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash c3adf078ce24a22166e51951a2d4b15f parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:25 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b065a60f-5939-461d-bdbf-2fea561316a6_characterizeImage_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b065a60f-5939-461d-bdbf-2fea561316a6_characterizeImage_947144_9.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2c8745ef-0e0f-48dd-a1cf-4a6651ed277d_isr_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2c8745ef-0e0f-48dd-a1cf-4a6651ed277d_isr_947144_9.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e97cb5101e098a68e70529b6c60c5c70 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 8df09fbf1e47dbbe7f130f9fa4efc5bb parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f391430e-f814-427c-8ae7-e993f6a757a1_characterizeImage_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f391430e-f814-427c-8ae7-e993f6a757a1_characterizeImage_947246_23.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/14e03d8e-33c8-4bd3-a24b-874d82c6dbdf_isr_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/14e03d8e-33c8-4bd3-a24b-874d82c6dbdf_isr_947246_23.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80675682b4d30439746cb5258f411310 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 53d94fcf77a491c473370044db32ac31 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/4f483d5c-7b62-4e69-8649-5d0ea74d9546_characterizeImage_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/4f483d5c-7b62-4e69-8649-5d0ea74d9546_characterizeImage_947140_39.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0f7e3634-d820-416b-bd96-06f4a7ac7f05_isr_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/0f7e3634-d820-416b-bd96-06f4a7ac7f05_isr_947140_39.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b4413a0fbfc15c97784b3760ac50a94 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash a050f33229ffc1453e057ef560163afb parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/1f8b188f-6914-4daf-bde4-05f80882feb8_characterizeImage_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/1f8b188f-6914-4daf-bde4-05f80882feb8_characterizeImage_947142_10.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fee92ddb-ae21-477e-82d7-d5baeb37ae32_isr_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/fee92ddb-ae21-477e-82d7-d5baeb37ae32_isr_947142_10.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f804d21385d8882054d7d5e73c7d0ab7 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 6723c575ebad82cb12b769697ef6251a parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcd12329-57ce-43f6-ba6b-cea195042a21_characterizeImage_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcd12329-57ce-43f6-ba6b-cea195042a21_characterizeImage_947142_12.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/975358d6-efa8-453d-b7ab-6f0b0ad3c8c6_isr_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/975358d6-efa8-453d-b7ab-6f0b0ad3c8c6_isr_947142_12.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 477fc84967f67ee4272b3cebc5f0b7e0 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 6b67c07f3551a336b1e335d38340ddf7 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f3ed0f45-cc4a-4266-b605-c73c95fb43c0_characterizeImage_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f3ed0f45-cc4a-4266-b605-c73c95fb43c0_characterizeImage_947140_5.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/dc343157-53ee-49ec-b778-c6da90b7bb6e_isr_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/dc343157-53ee-49ec-b778-c6da90b7bb6e_isr_947140_5.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0d0a198571b4fe052c1c46239318593 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 3d0fd4fa324eb709da448923adda9cf1 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f137adcf-a41b-4da2-ace6-b0d99ee47012_characterizeImage_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f137adcf-a41b-4da2-ace6-b0d99ee47012_characterizeImage_947140_12.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b0b7af53-c2e5-4f37-b9da-b290b7b968d2_isr_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b0b7af53-c2e5-4f37-b9da-b290b7b968d2_isr_947140_12.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24c01cc0c206e177f6708545f71bf3f2 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 40910a202537a35cda9163e2debb580f parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ecff300b-aae7-4929-82dd-a1ca5734d6af_characterizeImage_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ecff300b-aae7-4929-82dd-a1ca5734d6af_characterizeImage_947144_42.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a5c42b46-116b-467b-bde3-715d055c368d_isr_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/a5c42b46-116b-467b-bde3-715d055c368d_isr_947144_42.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d420ec71583b8847a7cd1744792d2a1 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash dca3513b7ba4a6824630600fe8120d0e parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e28f692b-5c2a-462f-8290-ef5b6aede70c_characterizeImage_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e28f692b-5c2a-462f-8290-ef5b6aede70c_characterizeImage_947140_36.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2412d4a4-c610-454e-80da-3754f295582c_isr_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2412d4a4-c610-454e-80da-3754f295582c_isr_947140_36.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 098d809994c501a6b5de07cf9f61b613 with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 1510ac395a5df2c487cc03d9ed2fd425 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a3804d6d-af6d-4c13-bf1a-caad4fa23345_characterizeImage_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a3804d6d-af6d-4c13-bf1a-caad4fa23345_characterizeImage_947142_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/913b01e4-340e-4943-9d1e-069cbb94e53b_isr_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/913b01e4-340e-4943-9d1e-069cbb94e53b_isr_947142_29.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0bad686f4ad0b58f1826ab49cce067 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 0422cefcf2b34bca3da598ff2c45bcca parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0181f7bc-e344-48c0-aa0e-5cdc99835f56_characterizeImage_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0181f7bc-e344-48c0-aa0e-5cdc99835f56_characterizeImage_947140_3.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e73544fb-b7ab-496b-bd2c-a9c60d7fa70f_isr_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/e73544fb-b7ab-496b-bd2c-a9c60d7fa70f_isr_947140_3.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311664343d3b491f9962637750d758c2 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 14eb5b6a1a33ac2b401fe230ef056764 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/08fb2935-6e8a-46bb-962f-d14bd7388bba_characterizeImage_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/08fb2935-6e8a-46bb-962f-d14bd7388bba_characterizeImage_947244_44.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6e51a0f1-98a2-4ce8-b1cc-334dd3d2a35e_isr_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6e51a0f1-98a2-4ce8-b1cc-334dd3d2a35e_isr_947244_44.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 777d8874598f3f0af5de0b9ed8f53c86 with result from task 789 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:27 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash a5304088d20d5bef418fa7bdfcaf65bb parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d146cbd1-f4d9-4d69-b193-6511a1d7aa38_characterizeImage_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d146cbd1-f4d9-4d69-b193-6511a1d7aa38_characterizeImage_947248_36.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3a59f128-32a2-42a6-989f-2c09257c809f_isr_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/3a59f128-32a2-42a6-989f-2c09257c809f_isr_947248_36.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f957f275772332001a3599b053ae9aff with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 86ef4c693a080f5f08b770d9eadf6673 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c87a454-2385-42b3-8b62-74271ca4bb2d_characterizeImage_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c87a454-2385-42b3-8b62-74271ca4bb2d_characterizeImage_947244_3.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f7e1bdc4-5082-4395-87ab-3490ea31a102_isr_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f7e1bdc4-5082-4395-87ab-3490ea31a102_isr_947244_3.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546796494c5469468d406a73def2a635 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 5dc737d001599680be95cbfec539b999 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/eca00987-368d-4905-914d-6303ed269e66_characterizeImage_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/eca00987-368d-4905-914d-6303ed269e66_characterizeImage_947142_18.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ae7b31db-6d01-4d8a-8c8d-8c4d4fdb015c_isr_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ae7b31db-6d01-4d8a-8c8d-8c4d4fdb015c_isr_947142_18.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f20d73230298a5c949aeae0048bdca with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash b014200b1d4bd3b09f6b367b1d249975 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d699e39e-6ab5-4547-bb25-6f6c915b26af_characterizeImage_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d699e39e-6ab5-4547-bb25-6f6c915b26af_characterizeImage_947248_54.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cb99c040-07d2-45a1-9582-ec5da6f488b5_isr_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/cb99c040-07d2-45a1-9582-ec5da6f488b5_isr_947248_54.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a539a7304d2f464e188fd88ef7423e1 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash eeaa8dc96851fb9b3c149cd6e8529649 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:27 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ff1e0be2-89ec-45e1-b4e1-0c6a3e2db220_characterizeImage_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ff1e0be2-89ec-45e1-b4e1-0c6a3e2db220_characterizeImage_947144_59.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f8927ca-e770-42c3-8f01-067792a69438_isr_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/7f8927ca-e770-42c3-8f01-067792a69438_isr_947144_59.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e7a038a2bb7a0af103922ddf77d79c8 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 36b72231db3c67d8c903617575e4c5f9 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/5645d668-fbe2-4d51-85a6-2d2937043b8d_characterizeImage_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/5645d668-fbe2-4d51-85a6-2d2937043b8d_characterizeImage_947140_32.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3413af2f-12a1-46b9-ab30-064437519f44_isr_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/3413af2f-12a1-46b9-ab30-064437519f44_isr_947140_32.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a09794392a1038b74a5ce712df42ea57 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 55a0db4b208169629651ce0a51c004c9 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/305313fa-00cf-4969-8cb3-eea84a06f412_characterizeImage_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/305313fa-00cf-4969-8cb3-eea84a06f412_characterizeImage_947246_17.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3deeabfd-29be-4b97-aefd-5fd1abe258b5_isr_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/3deeabfd-29be-4b97-aefd-5fd1abe258b5_isr_947246_17.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 967d67a8ff9c3a1f049a576ec1a518a1 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 13d0aa91ce7d867e1c1a7402d087bf2e parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/54f06075-a88b-483a-b646-c2b3aabe2115_characterizeImage_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/54f06075-a88b-483a-b646-c2b3aabe2115_characterizeImage_947246_29.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/040ad73d-1da1-4a31-b29a-9fc0cbc83bba_isr_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/040ad73d-1da1-4a31-b29a-9fc0cbc83bba_isr_947246_29.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ece016bcb5a27d879a5fa737411e2a0a with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash ae1d556bb214d1ad169032d562691d35 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/48c5cbfa-314d-4ed8-9746-2410273bbf59_characterizeImage_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/48c5cbfa-314d-4ed8-9746-2410273bbf59_characterizeImage_947244_32.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5c674f8e-b9f1-4018-a2e4-346c5dde4c51_isr_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5c674f8e-b9f1-4018-a2e4-346c5dde4c51_isr_947244_32.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e19f614f6ed890e7c63e2fbe05d12b38 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 0380520d8027bc8a1ae516df88888655 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/aee5a96e-ce53-4fe7-9d63-896cdfd8b1eb_characterizeImage_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/aee5a96e-ce53-4fe7-9d63-896cdfd8b1eb_characterizeImage_947244_59.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a6b093ef-ff6b-4382-a5eb-36e1d3ab08e2_isr_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a6b093ef-ff6b-4382-a5eb-36e1d3ab08e2_isr_947244_59.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b10389c36933fcedcab07e582b4030b with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash d65c61bf3b70affb1d2a0618243cb34d parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8a3fbcd9-7dc4-46d1-a150-3f699766aa65_characterizeImage_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8a3fbcd9-7dc4-46d1-a150-3f699766aa65_characterizeImage_947142_27.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4f0d9898-70c0-49b9-bc58-4717dc01ff4f_isr_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4f0d9898-70c0-49b9-bc58-4717dc01ff4f_isr_947142_27.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7356c3ab4957930aa7edf957cd6cb53 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 51a34e756a33051eb4bfa6c2ae3a7dc5 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/35d3e4c7-f275-4585-8ca9-5637a25c02b0_characterizeImage_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/35d3e4c7-f275-4585-8ca9-5637a25c02b0_characterizeImage_947140_22.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/ec277bd8-bde7-4777-8b26-6dcc2907d8a4_isr_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/ec277bd8-bde7-4777-8b26-6dcc2907d8a4_isr_947140_22.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d157e74cb7ac953d9264c56e8251431 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 7debec2f2541d87d5819d75f66444e52 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/291f38a6-e1b7-42fc-b200-44d91cd16ac8_characterizeImage_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/291f38a6-e1b7-42fc-b200-44d91cd16ac8_characterizeImage_947244_36.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/11659fff-092e-431b-a41c-86ad94c6cc97_isr_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/11659fff-092e-431b-a41c-86ad94c6cc97_isr_947244_36.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7bf1b10d6b488cbafce8e1677284faf with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 9337957960749982ccb7b6f0c6d1ec0f parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:28 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3e4c8f28-9dfb-4197-a3e9-890c2919676d_characterizeImage_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3e4c8f28-9dfb-4197-a3e9-890c2919676d_characterizeImage_947144_3.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8a388c9e-e7b2-4add-8763-df1086513668_isr_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8a388c9e-e7b2-4add-8763-df1086513668_isr_947144_3.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af553b133ea3fe979b664a0c95d89e4 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 9e78a0a3205cb972e21fcf3a20d962ec parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e51a6a3d-c1b7-4b3a-9bdd-0ae504b393c9_characterizeImage_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e51a6a3d-c1b7-4b3a-9bdd-0ae504b393c9_characterizeImage_947244_30.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/58382c53-700d-4462-b34d-7bf9ed91c25a_isr_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/58382c53-700d-4462-b34d-7bf9ed91c25a_isr_947244_30.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd2b2e47628bb44b2e10aff293a555eb with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 219a7eb65c568e3817a29c63f7680210 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5b292fce-7cca-4191-91d4-2128d947fb91_characterizeImage_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5b292fce-7cca-4191-91d4-2128d947fb91_characterizeImage_947244_46.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6af28d3-bdc9-4033-88da-36cab465c348_isr_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6af28d3-bdc9-4033-88da-36cab465c348_isr_947244_46.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c69ec2f60ffec5fcef1185296cc480c0 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash fb577c64e1176b3e4bc3303e44574425 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9942bff8-b065-47cd-87a8-3bae65a92a16_characterizeImage_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9942bff8-b065-47cd-87a8-3bae65a92a16_characterizeImage_947248_33.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/bddd4389-1e35-4fb3-8f4e-262b1691aef1_isr_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/bddd4389-1e35-4fb3-8f4e-262b1691aef1_isr_947248_33.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 725ccb04ff787da4074760982aa0b357 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 22de05d028962ef158b9fa1954093efa parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/89d37101-8c3e-46dc-8285-ce7aa7581fba_characterizeImage_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/89d37101-8c3e-46dc-8285-ce7aa7581fba_characterizeImage_947244_19.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/291cc40b-b95c-4caa-9f41-3ea3bf7cc734_isr_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/291cc40b-b95c-4caa-9f41-3ea3bf7cc734_isr_947244_19.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0f611c578d6cbf272bd04a577cbf52 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 06f47bd5e15f963f54dc6a6a2ee8b835 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a0f4434-c7bd-47a0-a2b5-273b7796f344_characterizeImage_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a0f4434-c7bd-47a0-a2b5-273b7796f344_characterizeImage_947244_42.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3523b5f6-9d08-42fa-a463-602de59a2571_isr_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/3523b5f6-9d08-42fa-a463-602de59a2571_isr_947244_42.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 082d6aebb1cf874e66035ababa09b800 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 8041bb5836e3a50ff38623ec09ce5fd0 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68791c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:28 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/4a3bd186-aa14-4779-8d29-8786a35f46cb_characterizeImage_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/4a3bd186-aa14-4779-8d29-8786a35f46cb_characterizeImage_947142_22.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8fc99916-0b64-4915-bb8c-f155c85e8c99_isr_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8fc99916-0b64-4915-bb8c-f155c85e8c99_isr_947142_22.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80b52a3d468c8497b980da275f98a5ee with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 42b57d072d4303c588e7a64d722f08af parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/df45d007-1503-42a2-afa3-0022c8a93b30_characterizeImage_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/df45d007-1503-42a2-afa3-0022c8a93b30_characterizeImage_947140_21.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/17074363-505a-4278-b903-f64cb87dd1a2_isr_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/17074363-505a-4278-b903-f64cb87dd1a2_isr_947140_21.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5949ea0ff33a9e2d584705416cbe9bcb with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 173ded12bc857f4691802ccc942c05b0 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:29 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7ee677c7-c3ab-4c1d-b4b0-d98eccb18eba_characterizeImage_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7ee677c7-c3ab-4c1d-b4b0-d98eccb18eba_characterizeImage_947248_32.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5a8bbb9c-da21-4ed8-ae5f-072a81fb35b6_isr_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/5a8bbb9c-da21-4ed8-ae5f-072a81fb35b6_isr_947248_32.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d5d8364a4e4f284e7e5d3bd66b1b92 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 3ac44aaeb03db17565df3011959589da parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c2cd7720-34a4-40cc-afcb-3e68b913a7be_characterizeImage_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c2cd7720-34a4-40cc-afcb-3e68b913a7be_characterizeImage_947246_25.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/87231100-8efc-4220-9d71-4ea26e433430_isr_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/87231100-8efc-4220-9d71-4ea26e433430_isr_947246_25.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59a50d5c2dc1bcd5c3877c855c717be6 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash bdb13fd5bcf6b494f2fa81b0fe8a6534 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c9d6b588-c571-4082-927a-a6160d612744_characterizeImage_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c9d6b588-c571-4082-927a-a6160d612744_characterizeImage_947140_46.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/50b96b63-644a-4f6c-ad8e-daf408b9c201_isr_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/50b96b63-644a-4f6c-ad8e-daf408b9c201_isr_947140_46.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f877229300a5b95d64bb29cbaa26f4f0 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 1995428beaf7c255d46d619e3e23421b parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:29 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5328ad17-c6a6-4446-a9f0-cac5c57786a7_characterizeImage_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5328ad17-c6a6-4446-a9f0-cac5c57786a7_characterizeImage_947142_25.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/eae7ac19-a8d7-4113-8a40-65534e3e5c58_isr_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/eae7ac19-a8d7-4113-8a40-65534e3e5c58_isr_947142_25.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93e9725324a93da0ed1a5737e60584e6 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash dcd77ae6c7502fdb215dafa197f2e053 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:29 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8735c15d-2c23-4048-b86c-e4780f4b73fa_characterizeImage_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8735c15d-2c23-4048-b86c-e4780f4b73fa_characterizeImage_947142_20.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4b9dc4c7-ca60-40a9-83e5-b0915f1ab013_isr_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/4b9dc4c7-ca60-40a9-83e5-b0915f1ab013_isr_947142_20.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22c2147aaf14cc5c8de928d1dcb04433 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 39d3b88751220709e4326e092cc7fd15 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:30 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/aea8f30a-3e00-4921-9653-5030e9b61ba1_characterizeImage_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/aea8f30a-3e00-4921-9653-5030e9b61ba1_characterizeImage_947142_49.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/6d928e6a-d79f-4c35-83cb-9d4911cb5b73_isr_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/6d928e6a-d79f-4c35-83cb-9d4911cb5b73_isr_947142_49.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8545b29608a3d2ece0be3ccd8a4986a with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 062fa3537fb08c68e0ba8aa84e49ca9d parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d0680c78-2af3-4d50-9493-0774cafeecb7_isr_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d0680c78-2af3-4d50-9493-0774cafeecb7_isr_947140_9.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72867c3822856112d25076dd5a446c00 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash df1c6ae4fcbdabb9532616da34d26e98 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8bc3643a-4c89-4759-b9e0-e3b204cc0c9b_characterizeImage_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8bc3643a-4c89-4759-b9e0-e3b204cc0c9b_characterizeImage_947140_11.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/aedb9997-15eb-4a46-95aa-1b2c0bc08330_isr_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/aedb9997-15eb-4a46-95aa-1b2c0bc08330_isr_947140_11.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81ed30d4ee7eaf859a6961ed7016b777 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 9e0b3b580623d7afd22d41a2761a7165 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9a015c11-9484-47dc-b800-b08ef12737af_characterizeImage_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9a015c11-9484-47dc-b800-b08ef12737af_characterizeImage_947142_52.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3be68ef7-646a-4bd6-9d08-ac7edb0373a0_isr_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/3be68ef7-646a-4bd6-9d08-ac7edb0373a0_isr_947142_52.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7b89c7a6309bf5dcd3f0aaae7867b9e with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 69c01d34e4e08e339766265f9456eaba parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e64685b9-1795-463a-ab57-e1c39bb667db_characterizeImage_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e64685b9-1795-463a-ab57-e1c39bb667db_characterizeImage_947144_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/427390eb-fe14-4492-9f5c-954cbb2602cc_isr_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/427390eb-fe14-4492-9f5c-954cbb2602cc_isr_947144_15.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8588acddebffdaba42c5b4a0119470 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash dfba2577e0367ac17b49dab803049c59 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:31 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/349e00a5-5675-481c-aeeb-149b69b120d3_characterizeImage_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/349e00a5-5675-481c-aeeb-149b69b120d3_characterizeImage_947144_37.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/de354ad1-9735-43d1-bc88-f8a0ae2e565d_isr_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/de354ad1-9735-43d1-bc88-f8a0ae2e565d_isr_947144_37.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a689d11e4cfc601aadc5d4bd6a7c0bd with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 59fe52c66cbb24a1c2f315cc05d1b1ab parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68787c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93cf55d5-6348-422c-8378-04d2a53c0bac_characterizeImage_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93cf55d5-6348-422c-8378-04d2a53c0bac_characterizeImage_947244_55.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4aed5285-f529-4e2d-a417-783181998bf6_isr_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4aed5285-f529-4e2d-a417-783181998bf6_isr_947244_55.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbd66690f5e89b4782e278e896ca016d with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 9885d444eda10e853dac07a7ec076db5 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a618a2bf-2f6e-4a26-991f-568acf8c13f9_characterizeImage_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a618a2bf-2f6e-4a26-991f-568acf8c13f9_characterizeImage_947246_54.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/76af494d-e3bf-42c3-aaee-038b7fdbac68_isr_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/76af494d-e3bf-42c3-aaee-038b7fdbac68_isr_947246_54.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a597226f9b66bc15a2b8acc8d2dccba with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 7e093a9fb313e2ef82b6d7d645bd8f83 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19cca926-4ca5-4359-b550-35b03501e94c_characterizeImage_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19cca926-4ca5-4359-b550-35b03501e94c_characterizeImage_947140_38.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c69c67c8-910d-4dc0-b338-f1a98f8626a8_isr_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c69c67c8-910d-4dc0-b338-f1a98f8626a8_isr_947140_38.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fada3335da369abc8b234a16c7ad056e with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 55c13bb8552ab4aed5142ae4f11583c5 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/41b43010-0df3-4b96-8ac9-83ec47f8a49d_characterizeImage_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/41b43010-0df3-4b96-8ac9-83ec47f8a49d_characterizeImage_947244_4.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a38a6401-41fc-4c10-a06c-6929a9f0a453_isr_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/a38a6401-41fc-4c10-a06c-6929a9f0a453_isr_947244_4.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b84288950832b61903fffb3c9291817 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash abe61c0817546d4d5ec8348dfe47c914 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4b42e88d-b0d5-4009-ab4d-919b423f9e7b_characterizeImage_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4b42e88d-b0d5-4009-ab4d-919b423f9e7b_characterizeImage_947248_35.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/e127fcdf-ac85-43cf-a1bf-4eed55bdfed5_isr_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/e127fcdf-ac85-43cf-a1bf-4eed55bdfed5_isr_947248_35.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72487c078c732d538b63dac32a86582 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 655bfe4fcc0dda81cf667504ebf481e9 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcac8c1a-a240-4c18-a779-b9263e35ada5_characterizeImage_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcac8c1a-a240-4c18-a779-b9263e35ada5_characterizeImage_947142_53.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7d592577-0de4-48d0-b3a4-85ad5f00c41d_isr_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/7d592577-0de4-48d0-b3a4-85ad5f00c41d_isr_947142_53.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f2944b69034b3e4335ded59476407f1 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash d4eb5571f9c0b050791c6c32450017d0 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/809118ae-0d5e-4f3f-ad99-5f437d00ddba_characterizeImage_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/809118ae-0d5e-4f3f-ad99-5f437d00ddba_characterizeImage_947248_52.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/68fbb37f-0792-4074-a831-fa99e86b19a0_isr_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/68fbb37f-0792-4074-a831-fa99e86b19a0_isr_947248_52.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0b2394ba4930f552a02fedab75d167 with result from task 720 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:31 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash ef644e9f2394c41444852464bbda0389 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8bf9b82f-0292-4e4e-8bd7-6b0495794cce_characterizeImage_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8bf9b82f-0292-4e4e-8bd7-6b0495794cce_characterizeImage_947248_29.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fb001287-5fa3-4bb3-a1fc-cbdd2536877c_isr_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/fb001287-5fa3-4bb3-a1fc-cbdd2536877c_isr_947248_29.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d77eeaa53372dbb5dd39a188d7d8f329 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 57e6c9ef0709d4226c5102872fe7b1d2 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c587049-ce14-447b-929c-a344b8e178fa_characterizeImage_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c587049-ce14-447b-929c-a344b8e178fa_characterizeImage_947244_31.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8e9826b5-f085-4392-86c6-65879739b837_isr_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8e9826b5-f085-4392-86c6-65879739b837_isr_947244_31.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25e6e8c637959f0c37cb5b2c9911b297 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash c601300932a6160afc30ea2411036c99 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/826a301c-9300-43c2-a6d7-2964d1928508_characterizeImage_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/826a301c-9300-43c2-a6d7-2964d1928508_characterizeImage_947140_29.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f9f7925a-1676-410c-83c1-ab56f85d4c38_isr_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f9f7925a-1676-410c-83c1-ab56f85d4c38_isr_947140_29.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8532686f0d44faa8786c182b5789ddbf with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash be7bdcdbfa06c6bd95f5ab961204e990 parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f61727e2-050e-403a-8e94-de7ca4025c18_characterizeImage_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f61727e2-050e-403a-8e94-de7ca4025c18_characterizeImage_947248_16.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/03a295b5-a469-4647-8ec9-e2443c244a9e_isr_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/03a295b5-a469-4647-8ec9-e2443c244a9e_isr_947248_16.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e4ecf727154a5386fe4f32e975146d with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 7299f76e62716f68743fd796440cb107 parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/305d708b-19bc-4622-a77e-638b07a9d320_characterizeImage_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/305d708b-19bc-4622-a77e-638b07a9d320_characterizeImage_947140_44.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f5cbd07-3c53-4a0e-bd5a-f6d328500ecd_isr_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/6f5cbd07-3c53-4a0e-bd5a-f6d328500ecd_isr_947140_44.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe00bd2c00ca2210d036dc591f26482 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 42190a9730f46c857f9e0e9a2f666d31 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19eb03d1-d253-47c0-b69a-c1038ac1032c_characterizeImage_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19eb03d1-d253-47c0-b69a-c1038ac1032c_characterizeImage_947140_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f4a207d1-eb59-4d78-9bbe-d9a6d0539b6b_isr_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/f4a207d1-eb59-4d78-9bbe-d9a6d0539b6b_isr_947140_16.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e38351386f2bc1cf333208086e79f5b5 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash ba226a8a3269bf9f51bc25f6ef49abb3 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e0289bf6-21b5-4eba-8c19-98b394706d11_characterizeImage_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e0289bf6-21b5-4eba-8c19-98b394706d11_characterizeImage_947244_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/84125a66-090c-4550-b533-818805b4dcc2_isr_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/84125a66-090c-4550-b533-818805b4dcc2_isr_947244_40.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b3b357873b10f09f6d7f92f2b0fec83 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 17c242769aef7946ee4c9e067166e194 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/20b9473b-0898-4bd8-b17a-3dc599593502_characterizeImage_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/20b9473b-0898-4bd8-b17a-3dc599593502_characterizeImage_947144_61.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/dd28e67e-a2dd-4856-b7a5-fb2e1eec81b8_isr_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/dd28e67e-a2dd-4856-b7a5-fb2e1eec81b8_isr_947144_61.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219c4fecb5caf679651eac6986ea6727 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 08fd4fc8961b8a8fbc8df45517979cb1 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3ee9c111-f99e-4937-b8ac-4f15d1a3d2e8_characterizeImage_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3ee9c111-f99e-4937-b8ac-4f15d1a3d2e8_characterizeImage_947142_13.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/19e37b87-bc99-4fef-87c3-f3984826f6b0_isr_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/19e37b87-bc99-4fef-87c3-f3984826f6b0_isr_947142_13.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d6c317216fdba2b05b1a4bc02f34058 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash a5c4791033829c22f7b53207837f7b9c parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/711e4a7a-a7a0-45e6-b04c-19889170dac2_characterizeImage_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/711e4a7a-a7a0-45e6-b04c-19889170dac2_characterizeImage_947244_18.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/19a51520-4b31-423f-a7be-a5801631e41e_isr_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/19a51520-4b31-423f-a7be-a5801631e41e_isr_947244_18.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b6657f00b6dd089ebaa960121599b2 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash b7ae0c6c89b266e30c3e2b44172fb569 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/bc1cad8e-81c3-4df7-8f9b-354a48532fac_characterizeImage_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/bc1cad8e-81c3-4df7-8f9b-354a48532fac_characterizeImage_947144_45.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/12493d1e-22c8-4501-9b74-37f9212f1e19_isr_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/12493d1e-22c8-4501-9b74-37f9212f1e19_isr_947144_45.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9910d60dc22f208bb7da6bdd5207b63 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 1527d3615ef77a7dddda06fd3d5814ae parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:32 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8f1cb8cf-c7e4-4594-ad46-8cd0d79f95ff_characterizeImage_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8f1cb8cf-c7e4-4594-ad46-8cd0d79f95ff_characterizeImage_947248_11.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/414b6a08-a3e3-4385-ab97-b96d7502b1d8_isr_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/414b6a08-a3e3-4385-ab97-b96d7502b1d8_isr_947248_11.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2131844d9447a3d3f01ad3d609bf4d with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 7e04fd45a51719e51ffee84b04cfd70b parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/75691a43-86dc-48ce-81af-a2317213c453_characterizeImage_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/75691a43-86dc-48ce-81af-a2317213c453_characterizeImage_947248_41.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/432360be-f90f-46f8-8dbb-d49c5cd0e63c_isr_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/432360be-f90f-46f8-8dbb-d49c5cd0e63c_isr_947248_41.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b04dd6a834331fdaf38cef588586a24 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash dd0fe128166447f688a87515e0d5a7ed parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/041adb41-1524-409c-b26f-43c927c842e7_characterizeImage_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/041adb41-1524-409c-b26f-43c927c842e7_characterizeImage_947244_20.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/41b62492-5a45-4612-be97-28ab489c9587_isr_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/41b62492-5a45-4612-be97-28ab489c9587_isr_947244_20.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80e245f357da91b9652e9d78b96ac90f with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 83101dc5f0e188417bc354e60ab6f865 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/c2775a9c-05b3-4f7e-82e0-28b99f1e6e1f_characterizeImage_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/c2775a9c-05b3-4f7e-82e0-28b99f1e6e1f_characterizeImage_947144_30.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/6ad6819d-8d66-43af-8e80-73f93259c85f_isr_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/6ad6819d-8d66-43af-8e80-73f93259c85f_isr_947144_30.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80048d9090b25d2801237da2b21bcd5 with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 1ec341faa6d143332510aaafb1f460f4 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d9458d79-5e6c-4ad5-ae17-c3753f204dd8_characterizeImage_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d9458d79-5e6c-4ad5-ae17-c3753f204dd8_characterizeImage_947248_9.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d00a7871-9807-43b5-8f7f-63a57bdec031_isr_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d00a7871-9807-43b5-8f7f-63a57bdec031_isr_947248_9.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d5b221de960af4fd052e16b4b219072 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash cdbd748f2978233de1aeaba97390845c parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0e5b8386-cd70-4527-8bbc-034dae6bed3b_characterizeImage_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0e5b8386-cd70-4527-8bbc-034dae6bed3b_characterizeImage_947244_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8a5188c5-6ca6-45c4-8b63-cc98f764c11b_isr_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/8a5188c5-6ca6-45c4-8b63-cc98f764c11b_isr_947244_43.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8e9f31e1896a8a0f86aedd4fb726350 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 96a0618a36df55270e9af2f1528934b7 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/dff21387-b017-473a-aced-975060b4f2f9_characterizeImage_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/dff21387-b017-473a-aced-975060b4f2f9_characterizeImage_947144_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/46b75715-6cbe-460d-9631-6cb9430e4ef2_isr_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/46b75715-6cbe-460d-9631-6cb9430e4ef2_isr_947144_26.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f7cbc01a6169c45cf40f6cbcdcab7b with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 705405b02f271d623a6c58b53433d5e3 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7df68f6b-2f1b-490b-a775-9408ddbb475f_characterizeImage_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7df68f6b-2f1b-490b-a775-9408ddbb475f_characterizeImage_947246_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d29f70f5-de9c-4d25-ac41-948bfecede10_isr_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/d29f70f5-de9c-4d25-ac41-948bfecede10_isr_947246_37.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9adaad61bc573ab2826f69d242c6122b with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 2366f4f091b69d8c6b8f35ddaff78c40 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/9649a5cf-79a1-4987-a45a-33c8e383fe0f_characterizeImage_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/9649a5cf-79a1-4987-a45a-33c8e383fe0f_characterizeImage_947140_42.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/18c2432c-f862-4f9a-baf1-c3d360ba086f_isr_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/18c2432c-f862-4f9a-baf1-c3d360ba086f_isr_947140_42.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6d453d295fad4d0bc85fe31c2277505 with result from task 810 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:32 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 48b491ad56fe715493fad0d9c4274ebf parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/448bf64b-36f4-404b-8d64-66787d89d7e8_characterizeImage_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/448bf64b-36f4-404b-8d64-66787d89d7e8_characterizeImage_947244_16.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/015d5a00-9360-4b3f-a129-98c69c0103f8_isr_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/015d5a00-9360-4b3f-a129-98c69c0103f8_isr_947244_16.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e333e541fe723a80cf2162774cf8dd69 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 937bee8fc6ca5de8bd773f54d02ea287 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c10ebcf-76d4-44cd-95eb-005a7ea698b5_characterizeImage_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c10ebcf-76d4-44cd-95eb-005a7ea698b5_characterizeImage_947244_62.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/baf044f8-20ef-4386-abf1-a6afc6d8c89f_isr_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/baf044f8-20ef-4386-abf1-a6afc6d8c89f_isr_947244_62.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5acf827ebe4df31b2e3682053b6dda36 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash d2c546c25d46601025a844c38c394ff8 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/80a9daf4-b3bb-40c7-9a29-4613b826d419_characterizeImage_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/80a9daf4-b3bb-40c7-9a29-4613b826d419_characterizeImage_947140_24.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d543f47c-28be-4e8a-a870-32eae17811a2_isr_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/d543f47c-28be-4e8a-a870-32eae17811a2_isr_947140_24.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed68398ece23fad0069e8cbb0deee7ea with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash ef3ce33efc968523168232c2266d186c parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/2b43e197-7e40-4a56-b074-1bd339e6e6fc_characterizeImage_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/2b43e197-7e40-4a56-b074-1bd339e6e6fc_characterizeImage_947244_60.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/31f35337-b9ed-4a4c-87a0-68b08b227ba6_isr_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/31f35337-b9ed-4a4c-87a0-68b08b227ba6_isr_947244_60.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70473224db55389c523753613982afa with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 6f742e6e189f8b5c1ca46eef62b1d231 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b10e417f-6381-4f5e-8837-607b7d95a3a5_characterizeImage_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b10e417f-6381-4f5e-8837-607b7d95a3a5_characterizeImage_947144_36.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/107b7c20-a87c-45a4-99c2-58517d5108e9_isr_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/107b7c20-a87c-45a4-99c2-58517d5108e9_isr_947144_36.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bc955373e967021a7afbe85825dabbe with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash f727c9baefb2cb7379999ac7216677f6 parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/00d9443d-4c67-4a75-a712-a8e176c91368_characterizeImage_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/00d9443d-4c67-4a75-a712-a8e176c91368_characterizeImage_947144_56.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0ac8c0c6-7e5d-463c-b1bd-068b1a62dc14_isr_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/0ac8c0c6-7e5d-463c-b1bd-068b1a62dc14_isr_947144_56.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2888c3178f37c1ad72ffff6d0b301a9b with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 154ed127983f4dbf62787582d5bca398 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f46b2ea1-1dc9-481f-bdfc-518f39e93176_characterizeImage_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f46b2ea1-1dc9-481f-bdfc-518f39e93176_characterizeImage_947246_59.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/98b4ba79-9a1a-41e6-a0a0-2648e217bd32_isr_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/98b4ba79-9a1a-41e6-a0a0-2648e217bd32_isr_947246_59.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65ff2deb6515d1abdd9b6f64f47da75e with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 6bceab9fb4e429edf6618caade8d3649 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3a2bb230-8936-4b97-810e-a8ea4621543f_characterizeImage_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3a2bb230-8936-4b97-810e-a8ea4621543f_characterizeImage_947142_57.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2fe457d3-0436-4317-873b-0e254f577531_isr_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/2fe457d3-0436-4317-873b-0e254f577531_isr_947142_57.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a64bc3f8b84ed5f629fe9af30a15d1d with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash c284e3f40ac0e587de6b2a43f9c05681 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c510443b-f296-4023-b45f-250e0d3c9237_characterizeImage_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c510443b-f296-4023-b45f-250e0d3c9237_characterizeImage_947248_50.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d9664ae4-1f1f-4555-adf0-ebb03d286323_isr_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/d9664ae4-1f1f-4555-adf0-ebb03d286323_isr_947248_50.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe0180c0c27e8a801b993c78a052e35c with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash dd75de12110bea256490fc5297c2b2a6 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/36aa0089-a77b-4e69-b975-8013157313fa_characterizeImage_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/36aa0089-a77b-4e69-b975-8013157313fa_characterizeImage_947142_36.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f996bb0b-e3dd-40e8-a185-1765148e0d0a_isr_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/f996bb0b-e3dd-40e8-a185-1765148e0d0a_isr_947142_36.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0cc8693dd21b19f7c37d053599c8c98 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 43f5fe2c968e67f67659cc951dadebd6 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eef7741e-d433-46b8-a432-711cde825175_characterizeImage_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eef7741e-d433-46b8-a432-711cde825175_characterizeImage_947244_54.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5a3717a6-7b6d-4210-a234-2bec1f8a156e_isr_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/5a3717a6-7b6d-4210-a234-2bec1f8a156e_isr_947244_54.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4abd5b19c58969ca377d021975d8b300 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 115ccbfa2f62886a30a6ec40ca24721e parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f01173c1-67fb-400b-bd27-86e01761d5bf_characterizeImage_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f01173c1-67fb-400b-bd27-86e01761d5bf_characterizeImage_947144_58.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8999eedd-6715-49b9-9beb-52a876d5472d_isr_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/8999eedd-6715-49b9-9beb-52a876d5472d_isr_947144_58.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7241ce7e24096a6e42b0b01191bb8a79 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 91e2761600c16eb5b241f37acd328db1 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:32 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cfb8e066-89c9-4ca6-85ed-c79148987f8c_characterizeImage_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cfb8e066-89c9-4ca6-85ed-c79148987f8c_characterizeImage_947140_59.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5f41e7da-d0e5-40ff-b5e9-2fcc9b8f84ff_isr_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/5f41e7da-d0e5-40ff-b5e9-2fcc9b8f84ff_isr_947140_59.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1074d23e47d445267829f8bc5bd11a73 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 87dde45bfdcab50e7ca6646d41512a2f parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5602e84e-e334-44b4-9270-4f0295ea31fd_characterizeImage_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5602e84e-e334-44b4-9270-4f0295ea31fd_characterizeImage_947246_24.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/8b538745-9fea-4ba5-932c-4c896748af39_isr_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/8b538745-9fea-4ba5-932c-4c896748af39_isr_947246_24.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fff3e0feca8ea81a19e27cd3eeffce15 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 7c3c217fab5140ed70d95b45bc715dfc parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/592560c0-365b-45b0-945b-7d207f161a8b_characterizeImage_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/592560c0-365b-45b0-945b-7d207f161a8b_characterizeImage_947144_52.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2a3c7934-bf77-434b-b495-8e55f91ec1fa_isr_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/2a3c7934-bf77-434b-b495-8e55f91ec1fa_isr_947144_52.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c65efea861b72c77dca139687db0bb7 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 6ea84bd96ae6eb664a993bc6ea25782a parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a4cec123-8613-4659-85f5-c65fefc3f6a1_characterizeImage_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a4cec123-8613-4659-85f5-c65fefc3f6a1_characterizeImage_947248_57.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9e97e117-8b9d-475d-9167-7051427b6173_isr_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/9e97e117-8b9d-475d-9167-7051427b6173_isr_947248_57.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f52419d830fb8bc647aeb43af18f852 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash cafe10fcb292d3017de4672221b7e2d1 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9a9efe67-2781-4b7c-a3d0-8109e1e3dc74_characterizeImage_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9a9efe67-2781-4b7c-a3d0-8109e1e3dc74_characterizeImage_947244_13.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6430f31-0561-445f-be2b-a7cc69e0e933_isr_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c6430f31-0561-445f-be2b-a7cc69e0e933_isr_947244_13.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83086d6059c2cadda9859d394af33c2 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash b2dfa46e909c52f70fa5b272dc3d2f23 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1ec4e725-ff89-4297-bc84-13b85c713e6c_characterizeImage_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1ec4e725-ff89-4297-bc84-13b85c713e6c_characterizeImage_947144_43.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d0513c1f-ea9d-48a7-9176-33afcc27beea_isr_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/d0513c1f-ea9d-48a7-9176-33afcc27beea_isr_947144_43.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4033ac9fab88cf9ab26dd0a90f7d207 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash edcc985e2a3641bddeb9aaa9adbecdd3 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/41492023-cc39-4ecd-9848-8ae647dab23a_characterizeImage_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/41492023-cc39-4ecd-9848-8ae647dab23a_characterizeImage_947144_14.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9cbb403f-4e0b-42ff-8863-046600e5a45b_isr_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/9cbb403f-4e0b-42ff-8863-046600e5a45b_isr_947144_14.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1272d3bf236b40f09a518c41e9108bb9 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 3f01d9f9c3eb5925003e524648908306 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4c104db1-a326-4816-b29d-90627f1809f9_characterizeImage_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4c104db1-a326-4816-b29d-90627f1809f9_characterizeImage_947144_55.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/c480c453-8489-41ed-bb76-e9e7a8c86cd3_isr_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/c480c453-8489-41ed-bb76-e9e7a8c86cd3_isr_947144_55.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d386db544230ad93f1abd502df248d82 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 1998e077258ce65b6cb428768658bdfb parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8af5e340-27af-41b6-8b81-53a409398981_characterizeImage_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8af5e340-27af-41b6-8b81-53a409398981_characterizeImage_947140_58.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/81f855f0-625f-4a0f-b351-ba9fc82103e2_isr_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/81f855f0-625f-4a0f-b351-ba9fc82103e2_isr_947140_58.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4d294cf09156d77f3f3dcc57ab8c4cd with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 17cc871b8f06384a586a46d95e41a20d parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b16784f5-f48c-42f1-a251-f76f1b3b0651_characterizeImage_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b16784f5-f48c-42f1-a251-f76f1b3b0651_characterizeImage_947246_42.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/296a9cc9-d726-4517-8c65-5c4b419b2df3_isr_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/296a9cc9-d726-4517-8c65-5c4b419b2df3_isr_947246_42.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0cfa233d6b4c37d69f5aa12c10d7cac with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 634264f3b90eac6b0eb6f26d1448d053 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2385e5a6-f900-4967-bf6e-e34665695709_characterizeImage_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2385e5a6-f900-4967-bf6e-e34665695709_characterizeImage_947142_4.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1f5b8af7-5759-4679-b85d-26a7160a82e9_isr_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/1f5b8af7-5759-4679-b85d-26a7160a82e9_isr_947142_4.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e284bb647ed9e736986139cf55668349 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash eb4520f86f78701a8e146f6ddfe27cec parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/64cef954-7e1c-4430-9cce-719f883377db_characterizeImage_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/64cef954-7e1c-4430-9cce-719f883377db_characterizeImage_947140_55.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b3872193-4bd4-4ec3-adf3-3427e826506c_isr_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/b3872193-4bd4-4ec3-adf3-3427e826506c_isr_947140_55.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f81e16e7eb191f7d89e0f7f499b3ab with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash b16745e9faa387667387255c46baad3b parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/32e54e32-4145-4347-a563-73640303cf98_characterizeImage_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/32e54e32-4145-4347-a563-73640303cf98_characterizeImage_947142_50.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af5ebc1f-03d1-4d26-b3e1-44ac3d82cd5b_isr_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/af5ebc1f-03d1-4d26-b3e1-44ac3d82cd5b_isr_947142_50.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bde2785b4c71ae119393cb2d3706eca7 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash accce9ec7b8f0e9b483672751dc30df6 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/00d612a2-fecc-4e0a-b52f-0fae14f8ab60_characterizeImage_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/00d612a2-fecc-4e0a-b52f-0fae14f8ab60_characterizeImage_947246_4.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b48e5ba3-cb44-4de2-8c11-6a7cba6aaf7b_isr_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/b48e5ba3-cb44-4de2-8c11-6a7cba6aaf7b_isr_947246_4.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dabf610ade9ebc455dd43746d4759c8 with result from task 954 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:32 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash dbea3054f66b3458dbdbb65b0371267a parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/b8a80675-120d-4e01-beb1-4da743c00ad8_characterizeImage_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/b8a80675-120d-4e01-beb1-4da743c00ad8_characterizeImage_947244_61.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4870ad0b-8565-4aa5-ab74-567441e799b0_isr_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/4870ad0b-8565-4aa5-ab74-567441e799b0_isr_947244_61.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718ff1c391de941f74b2a536c2afbdb9 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash 4de797befa17140595dce9f32797536c parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/638c986a-ef14-4933-9060-fdd130d71183_characterizeImage_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/638c986a-ef14-4933-9060-fdd130d71183_characterizeImage_947140_26.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2f7f8969-3b1f-425b-bdd1-48486ef26c07_isr_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/2f7f8969-3b1f-425b-bdd1-48486ef26c07_isr_947140_26.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8962ddc815f6265faeec4b113e986d with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 78c6529edf21b49b8a9555969f55e632 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/996fe5d2-9484-4796-8a55-c014ef874409_characterizeImage_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/996fe5d2-9484-4796-8a55-c014ef874409_characterizeImage_947142_48.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ac82363b-913d-432b-b437-e4287a56bd2c_isr_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/ac82363b-913d-432b-b437-e4287a56bd2c_isr_947142_48.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d5d6e0f1af6bb21d24385eaceb702c3 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 7c54a11ac66d5869c2858937caf78e0a parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8314b878-1242-49f9-8a2e-fdc5f3c16ed6_isr_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/8314b878-1242-49f9-8a2e-fdc5f3c16ed6_isr_947248_42.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82227befa00e3a418225cc36831ddcae with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 83bc95206ae25890fee6e5bbdd62fde3 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/c4cbe9b5-eaed-4d69-9dd8-19a287fb8eb4_characterizeImage_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/c4cbe9b5-eaed-4d69-9dd8-19a287fb8eb4_characterizeImage_947244_11.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f198ccfa-0142-4cee-819a-b2b5d1803719_isr_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f198ccfa-0142-4cee-819a-b2b5d1803719_isr_947244_11.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d407640a36dcfba619eac8ffa1abba93 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash b9650a8f56c781fcd2c766f0ce52f23f parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f06de00d-ebef-491a-aca1-b62477ce3f17_characterizeImage_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f06de00d-ebef-491a-aca1-b62477ce3f17_characterizeImage_947140_15.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a8c30454-3dcd-4e7c-895d-bbb6351fdad7_isr_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/a8c30454-3dcd-4e7c-895d-bbb6351fdad7_isr_947140_15.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3639e7aaf84745c4b0e6b738118972d7 with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash b4414f5ae04031078f039c3540194ae7 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/883c75a7-f2e8-4ebd-afb3-6313be83ba88_characterizeImage_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/883c75a7-f2e8-4ebd-afb3-6313be83ba88_characterizeImage_947244_27.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6ec75fa0-a1dc-4eb4-a819-d7b21ba3d141_isr_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/6ec75fa0-a1dc-4eb4-a819-d7b21ba3d141_isr_947244_27.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd1e1d78254c0cc5385f83312e2b6d1e with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 9aa06a468fd9bdb622f1088dbcf5382f parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e6ccf20d-fb79-4cae-949f-691ba5c7aeea_characterizeImage_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e6ccf20d-fb79-4cae-949f-691ba5c7aeea_characterizeImage_947244_58.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3fbacf7-d3b1-4537-8936-9ad374528d1c_isr_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/b3fbacf7-d3b1-4537-8936-9ad374528d1c_isr_947244_58.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f38cc47ceb31133cd52c3ac5a890e9b with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash dc2562e285403bf8e668f290b7cda223 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d16653f3-488d-41e2-b62c-25067e2e57db_characterizeImage_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d16653f3-488d-41e2-b62c-25067e2e57db_characterizeImage_947246_61.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1bd158da-3381-4183-b205-4233de44aaee_isr_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/1bd158da-3381-4183-b205-4233de44aaee_isr_947246_61.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83a1433112d1d4c373afd6276496d885 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash f90cfdff87e00ff246e68340692c541e parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/835dceee-4308-4813-be50-39ba3d4f5e10_isr_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/835dceee-4308-4813-be50-39ba3d4f5e10_isr_947144_2.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6df9f9aae8d78acfb7f780ed16fd44e9 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 4f3a7c1a05b2f9aafb18c82ff306226c parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:32 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/ba7c84e7-6ddf-4f90-bfd1-97db2ea18307_characterizeImage_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/ba7c84e7-6ddf-4f90-bfd1-97db2ea18307_characterizeImage_947246_12.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/da6b4f0f-dd4c-4498-8ae0-714ee1900a2c_isr_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/da6b4f0f-dd4c-4498-8ae0-714ee1900a2c_isr_947246_12.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da3256363a968e15516363970da84876 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 66cdb0a89a48c9022aeb0e2246c9f38f parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8cc49a56-c984-49f6-b1f2-231c6b53afd6_characterizeImage_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8cc49a56-c984-49f6-b1f2-231c6b53afd6_characterizeImage_947140_35.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/620f54bf-5e3f-45b1-ba66-1c20f38c8172_isr_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/620f54bf-5e3f-45b1-ba66-1c20f38c8172_isr_947140_35.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cede873dcc037dabdf375bbb2a148e1 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 3c0bf887063c9ca8a6f661f00269caac parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3f0218bc-d2c6-4052-ad3d-a2db49249086_characterizeImage_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3f0218bc-d2c6-4052-ad3d-a2db49249086_characterizeImage_947248_62.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47b4174d-cd11-4840-b9ee-454106a85536_isr_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/47b4174d-cd11-4840-b9ee-454106a85536_isr_947248_62.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19eefae961224b300fee5892888fa7b with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 80d55103ac8dd4b09dbec381c13713b2 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ffc6396c-30d3-48fb-96cb-31e7bcb7f9b3_characterizeImage_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ffc6396c-30d3-48fb-96cb-31e7bcb7f9b3_characterizeImage_947244_41.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63c6422f-d002-41ff-96b7-9c7934124716_isr_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/63c6422f-d002-41ff-96b7-9c7934124716_isr_947244_41.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e4b24d714969f203163326d28fc49d with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash ce17e5c5918614d645caecd817d2fbc5 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/21f73a23-6a44-4c3c-a22b-cb41142d32a7_characterizeImage_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/21f73a23-6a44-4c3c-a22b-cb41142d32a7_characterizeImage_947244_12.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cde92c-7aa2-4030-add1-56acddbda51f_isr_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/c8cde92c-7aa2-4030-add1-56acddbda51f_isr_947244_12.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b24fa3ba9028760cb775d3deefc36a01 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash 7f3221235df861a407080930155b0893 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2b10af11-c35b-4411-8ceb-ccb0e6fb9df5_characterizeImage_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2b10af11-c35b-4411-8ceb-ccb0e6fb9df5_characterizeImage_947248_34.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4af2aa3a-fea4-424b-81e7-a8eff2d19c65_isr_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/4af2aa3a-fea4-424b-81e7-a8eff2d19c65_isr_947248_34.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84865613e457d4722c716ae0ee87ac0 with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 4c1d1eb06cf2aaeaed94c64729ad64dd parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9a3e8239-f487-41b1-b90e-6d90c10ded10_characterizeImage_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9a3e8239-f487-41b1-b90e-6d90c10ded10_characterizeImage_947248_46.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/10c97cad-771b-4592-ba99-f309670bf25d_isr_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/10c97cad-771b-4592-ba99-f309670bf25d_isr_947248_46.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e55cc8b612a5d0bdfbc4d8340e5c175d with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash d1ea49b71ee50469c6cae3c4485e91cf parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8d1ce7c1-bdd7-4fb1-ae74-f7a7dfee810a_characterizeImage_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8d1ce7c1-bdd7-4fb1-ae74-f7a7dfee810a_characterizeImage_947140_48.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/64bfd8ce-e1a8-41d8-9b51-239b63311823_isr_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/64bfd8ce-e1a8-41d8-9b51-239b63311823_isr_947140_48.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7b6eb9b8e84b702b5cf3a30ec9d47af with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 8a71733e6d59445e05f56f81629ecaf7 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:33 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/1790380c-0ec5-47cd-890b-31cc3dad5416_characterizeImage_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/1790380c-0ec5-47cd-890b-31cc3dad5416_characterizeImage_947140_56.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c589dcef-cf3a-43f4-b05c-15dd97c056f8_isr_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/c589dcef-cf3a-43f4-b05c-15dd97c056f8_isr_947140_56.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc8dbf7696b7c192a031dbdeb5c999b with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 5887cf56777e81da6dd24bfbf51f8731 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bbb8763d-5f37-4e60-b07b-7a0cb52b3649_characterizeImage_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bbb8763d-5f37-4e60-b07b-7a0cb52b3649_characterizeImage_947142_16.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8c5ce78e-efdf-4677-9a62-3613d83912ef_isr_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947142/8c5ce78e-efdf-4677-9a62-3613d83912ef_isr_947142_16.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426d967fa8e4af4e3f74c18489b00849 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash b625d5c00a249572c1eb4875474e3614 parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/37222aeb-bb01-43b8-9531-57790485480d_characterizeImage_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/37222aeb-bb01-43b8-9531-57790485480d_characterizeImage_947246_20.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/913454f7-a3c2-47e8-a7f2-bc3312677f7c_isr_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/913454f7-a3c2-47e8-a7f2-bc3312677f7c_isr_947246_20.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae6b0a260bae436e47e1f4fdb6c5b90 with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash da5d3ffe96326a99216c898785e9bfaa parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/971e205c-994a-49b1-b054-1f2f4ab31b7b_characterizeImage_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/971e205c-994a-49b1-b054-1f2f4ab31b7b_characterizeImage_947246_31.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c03115f2-027c-4fba-85da-e2c563fa08df_isr_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/c03115f2-027c-4fba-85da-e2c563fa08df_isr_947246_31.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f386766aced367d89f1800bab88c80a with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 7765451958e22ffbd9ed4848351d1279 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/13d52003-6524-4a43-9cc5-df8ae91884b4_isr_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947248/13d52003-6524-4a43-9cc5-df8ae91884b4_isr_947248_51.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4126ca1559a15a87149c3fe836f440fe with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 65789c1e6eca2236b1fd23b58590fe10 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f6be0c1f-685b-4bbe-8b03-bcc47acbc8b8_characterizeImage_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f6be0c1f-685b-4bbe-8b03-bcc47acbc8b8_characterizeImage_947246_28.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/84f1bf39-b1f0-4706-bab2-5f69c8038f02_isr_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947246/84f1bf39-b1f0-4706-bab2-5f69c8038f02_isr_947246_28.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881bdcc41c86beae669405f902da204d with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 27d24797c3dc1804d4d6f2934badb271 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1e752b25-dfc1-40bf-bc50-2e574b62b7da_characterizeImage_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1e752b25-dfc1-40bf-bc50-2e574b62b7da_characterizeImage_947244_15.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45994b8-800a-4eb7-b938-8f4e5001e140_isr_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947244/f45994b8-800a-4eb7-b938-8f4e5001e140_isr_947244_15.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1375a72229f9cb1bac3113f5f08916a with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash db8e6040ae60b1db07a7b98a277d0e22 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c86ab917-1186-4849-81a6-7309a1ed6191_characterizeImage_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c86ab917-1186-4849-81a6-7309a1ed6191_characterizeImage_947140_40.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/905c0732-6424-4a0c-9a8a-0f36ea4df6d2_isr_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947140/905c0732-6424-4a0c-9a8a-0f36ea4df6d2_isr_947140_40.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d25d3b1bb1685dfbf28cd4a4a0cd9e with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 96748ef4287aeca3a79f38cec7a889c8 parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:33 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/829c01cf-883e-4a51-af3f-953e2f93ef1b_characterizeImage_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/829c01cf-883e-4a51-af3f-953e2f93ef1b_characterizeImage_947144_48.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/94aab505-0720-4cf5-9bbf-d903fddbcf69_isr_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/94aab505-0720-4cf5-9bbf-d903fddbcf69_isr_947144_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1096 try 0 failed parsl.dataflow.dflow INFO: Task 1096 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 1a7050a4d731f912cb8615601fcee24f parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 1 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e6a980bcc644107b1ced8f454fbe69 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash f331e11f9b9c626b64a4c71e2751182f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/11d2dd37-aacb-46c1-a418-df32638043dd_calibrate_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/11d2dd37-aacb-46c1-a418-df32638043dd_calibrate_947140_23.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/897a2906-2085-49cd-af33-548a9c936d71_characterizeImage_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/897a2906-2085-49cd-af33-548a9c936d71_characterizeImage_947140_23.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a199f2b78e233b9613f1dfbcd3cec7 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash bb5a7c6fcfced45d013d20a875708a33 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3a7ccf28-0f9b-418e-a1b8-c577bda7de6e_calibrate_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3a7ccf28-0f9b-418e-a1b8-c577bda7de6e_calibrate_947142_33.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b5b5cd0d-66f3-46b8-83fb-2caeaabd806e_characterizeImage_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b5b5cd0d-66f3-46b8-83fb-2caeaabd806e_characterizeImage_947142_33.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219b969594c2abfb1b9fe62bcb06ee11 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash e2e075e39bb9507a015e21d1197b93d9 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b59c451-8d66-434a-8301-b36d91865e7d_calibrate_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b59c451-8d66-434a-8301-b36d91865e7d_calibrate_947140_13.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c1f3d56f-ed39-4bbb-900d-7798f73af8a4_characterizeImage_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c1f3d56f-ed39-4bbb-900d-7798f73af8a4_characterizeImage_947140_13.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 623a7f260cf718d6a205fe8812286eab with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 6e10b227402599a2d0df803789a61d58 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b4ac2cb-8e3d-41ab-ae59-b41f848335c3_calibrate_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b4ac2cb-8e3d-41ab-ae59-b41f848335c3_calibrate_947140_57.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6066e577-1b11-4254-bb73-58de772a61d4_characterizeImage_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6066e577-1b11-4254-bb73-58de772a61d4_characterizeImage_947140_57.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2158056c973c37e7e8aa3cac10ce520 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 1770bf4b44a6ee2e7854d8fc4a6ae675 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b2fb7630-2f91-4b4f-8ff1-e806cc2be4b7_calibrate_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b2fb7630-2f91-4b4f-8ff1-e806cc2be4b7_calibrate_947142_14.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5e18a9e4-4238-4f49-95f1-946c59469ab0_characterizeImage_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5e18a9e4-4238-4f49-95f1-946c59469ab0_characterizeImage_947142_14.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f477df6f271645fc4e951d9d808ec8 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash fa9d3afb6b73f99c2f4fa292994290eb parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/28713079-9e9e-4445-bf18-c1d94b86eccf_calibrate_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/28713079-9e9e-4445-bf18-c1d94b86eccf_calibrate_947140_30.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/84e351fb-9e2d-498a-bd11-9d13cbf51373_characterizeImage_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/84e351fb-9e2d-498a-bd11-9d13cbf51373_characterizeImage_947140_30.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65da0795a1b6ca9ccbe2c4eb7dfda63d with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash c541ecfb903e52bcb6d5b9a97e82e4b1 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8d3074bf-7cfe-41d1-bafe-e385cae2b38b_calibrate_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8d3074bf-7cfe-41d1-bafe-e385cae2b38b_calibrate_947140_43.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a3a9602f-c1fc-4cbe-8ed0-c4632b834490_characterizeImage_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a3a9602f-c1fc-4cbe-8ed0-c4632b834490_characterizeImage_947140_43.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4360b5f96a50a3fbfaf603601779ae0 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 91db10fcd3ee18435114154915fdd078 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a497f62a-94c1-4848-85c4-5b5d86cee23f_calibrate_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a497f62a-94c1-4848-85c4-5b5d86cee23f_calibrate_947144_8.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ddfba5e5-ed84-4633-8ede-d6e7febfc6d3_characterizeImage_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ddfba5e5-ed84-4633-8ede-d6e7febfc6d3_characterizeImage_947144_8.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47f42fd227ac23982ae5d19796977b49 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 67b88d7a9bf35ab1335cc839217adfb2 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5eb212c1-2c5c-4c35-8b73-a643105fa387_calibrate_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5eb212c1-2c5c-4c35-8b73-a643105fa387_calibrate_947244_57.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eb0400d7-c930-44c6-8406-26070f78af02_characterizeImage_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eb0400d7-c930-44c6-8406-26070f78af02_characterizeImage_947244_57.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d163078f0c56c810c1188eb8e365f779 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 4e0643f98d1a11cb5604870dff8def46 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3139b048-d5f3-4903-af08-7f1232229b9f_calibrate_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3139b048-d5f3-4903-af08-7f1232229b9f_calibrate_947140_10.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2220a06e-6ce7-4f80-abc9-03adae6bdb00_characterizeImage_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2220a06e-6ce7-4f80-abc9-03adae6bdb00_characterizeImage_947140_10.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d44256025e07c2ef3c57110864f69e7 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash c1429d7a919069cbe4f517b57fa63b56 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/68b375b7-9386-4f0a-8c2d-a380d962800d_calibrate_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/68b375b7-9386-4f0a-8c2d-a380d962800d_calibrate_947142_58.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/fbdab855-0ed3-44a9-9ad9-93435cf13e82_characterizeImage_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/fbdab855-0ed3-44a9-9ad9-93435cf13e82_characterizeImage_947142_58.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f097abae778be0610a2aed3e7ffdfe with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 8d0d2b16d263d66e2a8fd7940a07fb6a parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9843b2f3-0521-4bc5-a294-57d89da6482c_characterizeImage_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9843b2f3-0521-4bc5-a294-57d89da6482c_characterizeImage_947142_41.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956d93133343d88e1c334ce2ffcc22cf with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash fe3ca3971d36385f1b1233174677e8c2 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:33 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/7b37012e-ef90-48d4-bf52-fefeab65d4e4_calibrate_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/7b37012e-ef90-48d4-bf52-fefeab65d4e4_calibrate_947144_23.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/eb04b945-9168-46a3-a094-705a4ac23b44_characterizeImage_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/eb04b945-9168-46a3-a094-705a4ac23b44_characterizeImage_947144_23.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fe5e23b2a56412d05e7ea5820f3a0b with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash dd896bd53b40563786fcffafa1f9f223 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a4acac06-671c-4ed1-b06a-fa728e66508c_calibrate_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a4acac06-671c-4ed1-b06a-fa728e66508c_calibrate_947142_3.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/7ca0aff0-e93f-403f-9e85-6e05c5f04728_characterizeImage_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/7ca0aff0-e93f-403f-9e85-6e05c5f04728_characterizeImage_947142_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 169 try 0 failed parsl.dataflow.dflow INFO: Task 169 marked for retry parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 15ecd42793d43aaa5e82231680c692e2 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 1 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e702a0de56c39a5a8b7d6c71305a01ac with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash ac0a49e766d737e7c61cace397d9e562 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/81b49969-4d65-45c7-a3d9-6dc4c843db89_calibrate_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/81b49969-4d65-45c7-a3d9-6dc4c843db89_calibrate_947140_49.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/79db0a9b-0e67-4ace-aed6-8ef45281cc49_characterizeImage_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/79db0a9b-0e67-4ace-aed6-8ef45281cc49_characterizeImage_947140_49.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de2acaab3f27980fcb77c1dab8a9cd3 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash d6263748383ec4c1b2eb0ce20ff9b797 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9214c3ef-560a-474b-b319-c9cc413a6127_calibrate_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9214c3ef-560a-474b-b319-c9cc413a6127_calibrate_947144_54.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4e504a77-fb5f-4498-87b7-9198b7dc37c0_characterizeImage_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4e504a77-fb5f-4498-87b7-9198b7dc37c0_characterizeImage_947144_54.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf11ece3f3c4beb0c1ddbd8084b00923 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 5e879e0c6930c89345d07f1a9c60a25b parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0de8c5da-770d-4212-8f83-862da84d97cb_calibrate_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0de8c5da-770d-4212-8f83-862da84d97cb_calibrate_947142_28.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/0992cadd-b0f5-4f8b-94c9-a6978beebbc7_characterizeImage_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/0992cadd-b0f5-4f8b-94c9-a6978beebbc7_characterizeImage_947142_28.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a613e1846c4113a251f443a39045fbfd with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash d81fc38b2476c1442122ad131d62df0b parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ffb4d8d1-fe87-4520-aef5-d4d16add3f18_calibrate_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ffb4d8d1-fe87-4520-aef5-d4d16add3f18_calibrate_947144_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/664c40a8-690c-4653-8cf8-ce21b1f35af1_characterizeImage_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/664c40a8-690c-4653-8cf8-ce21b1f35af1_characterizeImage_947144_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 571 try 0 failed parsl.dataflow.dflow INFO: Task 571 marked for retry parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash ffdde4c0e7cc88efec14383db4adddc4 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 1 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 128e35c5fb0ed7657c9d1630979439ba with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 404a3a41868c95d8e251c4445aea6367 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/331c3021-2564-4862-9e50-3f3e1053a757_calibrate_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/331c3021-2564-4862-9e50-3f3e1053a757_calibrate_947246_45.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7637263a-3c55-4029-a92f-26f8c9639beb_characterizeImage_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7637263a-3c55-4029-a92f-26f8c9639beb_characterizeImage_947246_45.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8868c53df714b25a92b17edc74963295 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 7aa788e7d3a41a489810fff83c51425f parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ba07c121-d5da-40b3-ae69-f68a76a69c79_calibrate_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ba07c121-d5da-40b3-ae69-f68a76a69c79_calibrate_947140_51.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/09fc6ca8-a463-4327-a29e-b9d33ee27c47_characterizeImage_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/09fc6ca8-a463-4327-a29e-b9d33ee27c47_characterizeImage_947140_51.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ba35635316bb8ce6abbb3a38b845c41 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash f7c257450d74316fdcf02521e46d1dac parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/56e5eae4-efd7-48c1-8ffe-26592430857b_calibrate_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/56e5eae4-efd7-48c1-8ffe-26592430857b_calibrate_947142_15.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/67b1a32f-05a7-4474-a9bf-e8f3379c418e_characterizeImage_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/67b1a32f-05a7-4474-a9bf-e8f3379c418e_characterizeImage_947142_15.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4762a4a00bf9868aebdb9647d6368465 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 0b4ff89723cd2d26df435f41d8dbe8e9 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1c213975-336c-4772-86f2-8d2c30d74ee7_calibrate_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1c213975-336c-4772-86f2-8d2c30d74ee7_calibrate_947246_11.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/134f01ae-197d-4d48-a9c9-ae64fe9dba33_characterizeImage_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/134f01ae-197d-4d48-a9c9-ae64fe9dba33_characterizeImage_947246_11.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b0ed97d4098fd5781c6395b1389d240 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 3ffd7e8c2c6dfcfe956c03dfa4aedc5c parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/69f093bb-e71a-4717-9952-4734a4dd3c81_calibrate_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/69f093bb-e71a-4717-9952-4734a4dd3c81_calibrate_947248_4.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5144c21e-d6fc-4b85-96da-ed5f3ced8bdf_characterizeImage_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5144c21e-d6fc-4b85-96da-ed5f3ced8bdf_characterizeImage_947248_4.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 481b88b37d8db89083f7f5da9965461c with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 02badb582ff9f4b2d8b082bf9290c2af parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/de282414-f679-4b22-9615-9c8014766248_calibrate_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/de282414-f679-4b22-9615-9c8014766248_calibrate_947144_35.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f9631638-6f4b-48c6-8941-c1faab0a08b9_characterizeImage_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f9631638-6f4b-48c6-8941-c1faab0a08b9_characterizeImage_947144_35.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ed862c169e8ab866c05f84ae1d4e3e with result from task 187 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:33 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 741030aa0c607d6a3b9b6dcdf9c9db76 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3844e23c-857f-486a-81eb-d2f739e68fbb_calibrate_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3844e23c-857f-486a-81eb-d2f739e68fbb_calibrate_947244_52.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/daeeaa24-c2b5-42c0-9030-bfc306690e48_characterizeImage_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/daeeaa24-c2b5-42c0-9030-bfc306690e48_characterizeImage_947244_52.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b5dafe718782c58435eabbb113e728 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f98769daa6c64b1dd656ee3497744bd0 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c495bae1-02d3-4d00-a561-ea6927c8bdcb_calibrate_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c495bae1-02d3-4d00-a561-ea6927c8bdcb_calibrate_947244_51.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/7e7d85bc-1963-498b-91c5-7bba76a863fa_characterizeImage_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/7e7d85bc-1963-498b-91c5-7bba76a863fa_characterizeImage_947244_51.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84d62c331594658beec2dd8e3c145ef3 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash e0f702673f8e796bc0526333f37823af parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1857a075-8860-45f1-bdfd-2f33c4a563dd_calibrate_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1857a075-8860-45f1-bdfd-2f33c4a563dd_calibrate_947246_60.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5f2807f2-9671-4181-a68e-cdcf5b94d0d5_characterizeImage_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5f2807f2-9671-4181-a68e-cdcf5b94d0d5_characterizeImage_947246_60.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa35657a505c6040e22341e545e288c8 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 0e8fb9ebce46d4baa9779b3dd51139b3 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8b2ed851-662c-49c1-a78f-4c188699cfcb_calibrate_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8b2ed851-662c-49c1-a78f-4c188699cfcb_calibrate_947246_48.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/619abb6c-3411-46ae-8bf2-be722cce9480_characterizeImage_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/619abb6c-3411-46ae-8bf2-be722cce9480_characterizeImage_947246_48.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6866af8c263a1d51fe5d21dda1f762b9 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c8084453cafbb836a2410671b398eddd parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/35adfc82-d3d1-49b7-83b1-a09358e4e621_calibrate_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/35adfc82-d3d1-49b7-83b1-a09358e4e621_calibrate_947248_48.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a94db-f10d-4346-896b-a063f981be2b_characterizeImage_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a94db-f10d-4346-896b-a063f981be2b_characterizeImage_947248_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 304 try 0 failed parsl.dataflow.dflow INFO: Task 304 marked for retry parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 247e7560f1f011c3aba1b0f695838e83 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 1 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9b2284aa93b6254f04f063bfcbad753 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 33cdec5157f7be836bad4c64db5a5e8d parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a77740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b149c64f-606f-4f91-91a3-4338e702b416_calibrate_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b149c64f-606f-4f91-91a3-4338e702b416_calibrate_947244_9.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a7690d4-da6d-404d-933a-cff913608f8d_characterizeImage_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a7690d4-da6d-404d-933a-cff913608f8d_characterizeImage_947244_9.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07f4c2740cd249ddca9240414dc33f68 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash ec5bf5a39c1352bbca675fd520e9717d parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08de91c5-6287-4ff5-a599-8cf216f4d307_calibrate_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08de91c5-6287-4ff5-a599-8cf216f4d307_calibrate_947248_60.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/6db06895-ef6b-452d-86c3-9c2bb63c08d3_characterizeImage_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/6db06895-ef6b-452d-86c3-9c2bb63c08d3_characterizeImage_947248_60.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18742ba57cf83ec3d6dea34ae29b7d31 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 09e46ce81b0634d6219ca1736b3e9fa7 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc0981a1-33fe-4378-ab51-ae2a2035c168_calibrate_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc0981a1-33fe-4378-ab51-ae2a2035c168_calibrate_947140_20.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/70a6f99b-4139-447a-abce-55f291f7726b_characterizeImage_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/70a6f99b-4139-447a-abce-55f291f7726b_characterizeImage_947140_20.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01e0b15c9a72cb9feb10ae28b9142e40 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 3e5e381853cacf675b80c9c13fdfa40a parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/455d8ce8-4d74-49dc-ad21-766b3b17e87e_calibrate_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/455d8ce8-4d74-49dc-ad21-766b3b17e87e_calibrate_947142_5.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d12a66bf-ffa0-4e16-81b9-a4bffc36581c_characterizeImage_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d12a66bf-ffa0-4e16-81b9-a4bffc36581c_characterizeImage_947142_5.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65dfe36ba0bd8987d12939d30f79b9d7 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash b98a088c5d85c50a1d608a6cc37346b9 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a97b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2c8bcced-80c2-48c2-bc9a-2590e36ec6e3_calibrate_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2c8bcced-80c2-48c2-bc9a-2590e36ec6e3_calibrate_947246_26.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e3c09037-4843-4ad0-b676-d5d020c7a585_characterizeImage_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e3c09037-4843-4ad0-b676-d5d020c7a585_characterizeImage_947246_26.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ebe723b6cc7367ecdea42521cecbf80 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash cc310fd40b923d0ad542dd95da92e2ae parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/fe2be2f5-7bc0-4dd7-8c14-e9745f21ce46_calibrate_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/fe2be2f5-7bc0-4dd7-8c14-e9745f21ce46_calibrate_947142_34.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/593d8d58-118e-4bea-8520-7e2fe26e7273_characterizeImage_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/593d8d58-118e-4bea-8520-7e2fe26e7273_characterizeImage_947142_34.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fda88171db8b7e594f90575e22a9b777 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash b389ee47ff311805733ba575e36cc8ff parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b301575d-1323-4122-a324-7090899c4832_calibrate_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b301575d-1323-4122-a324-7090899c4832_calibrate_947248_17.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/713019d4-699a-4726-90cc-a08dccaf804a_characterizeImage_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/713019d4-699a-4726-90cc-a08dccaf804a_characterizeImage_947248_17.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68dea4276cec3b4addc10115c1940c6c with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 4b85f4f22d707d9f17a9a6f92a09b250 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c49f566a-f305-4821-a4f5-4e5e14c845c6_calibrate_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c49f566a-f305-4821-a4f5-4e5e14c845c6_calibrate_947140_27.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/56b9f0f0-70db-4ce9-ad6d-aef6b5b65060_characterizeImage_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/56b9f0f0-70db-4ce9-ad6d-aef6b5b65060_characterizeImage_947140_27.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73662b0167b48207b278d9b8e48c382c with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 8becbbc6f437c96624f9417ebcd3e03b parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a286be51-e3a9-4822-98ad-9baf2a6b7075_calibrate_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a286be51-e3a9-4822-98ad-9baf2a6b7075_calibrate_947144_46.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8f603b60-8650-402a-be6c-69346a58dea6_characterizeImage_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8f603b60-8650-402a-be6c-69346a58dea6_characterizeImage_947144_46.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f2ef4cc9b32054447c0ef6f647e080 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 84fccc3768d4d864ac030dd50cf5cd50 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0fc905d9-72b2-4c7c-ad0c-b14335d5f812_calibrate_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0fc905d9-72b2-4c7c-ad0c-b14335d5f812_calibrate_947248_5.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a3bc7-0622-423d-bff4-42673c4b2194_characterizeImage_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bd5a3bc7-0622-423d-bff4-42673c4b2194_characterizeImage_947248_5.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2984ef128339f7468cc3f14beefb740 with result from task 1072 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:34 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 4f19facd227b206a2d196ded146d8bdb parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67faf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ac68a730-0c0d-48b9-8894-afcadc356080_calibrate_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ac68a730-0c0d-48b9-8894-afcadc356080_calibrate_947140_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/92d3e6cd-2b93-4ecc-a110-9c8857b9da9f_characterizeImage_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/92d3e6cd-2b93-4ecc-a110-9c8857b9da9f_characterizeImage_947140_52.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7bc8d9cb95615e8356cf1a445b1cfea with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 3d7875e35c3197f6135519b13667b52c parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e56b921c-50be-4b98-8fd6-a2abb94b0fc7_calibrate_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e56b921c-50be-4b98-8fd6-a2abb94b0fc7_calibrate_947142_59.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a433c596-319e-4c01-91dc-aff51bc32733_characterizeImage_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a433c596-319e-4c01-91dc-aff51bc32733_characterizeImage_947142_59.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 200ae794a8a05d487187cd924fcf00fc with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash f56ea74bc565b288b9fc2b7307508072 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a74ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/355aac0c-3bf8-4892-beec-fdecc7a5d9c9_calibrate_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/355aac0c-3bf8-4892-beec-fdecc7a5d9c9_calibrate_947144_5.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6646fde9-f5b0-4c30-ae0d-962aaea160d1_characterizeImage_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6646fde9-f5b0-4c30-ae0d-962aaea160d1_characterizeImage_947144_5.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cabd34c9c32a9403ecac05435491e6dc with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 3713eb33abde841a0ba05e5e29fb89ec parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b855ccd-7fac-4667-92cf-823c430b8939_calibrate_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b855ccd-7fac-4667-92cf-823c430b8939_calibrate_947140_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/983f2213-7d03-416b-88cb-381375d587f5_characterizeImage_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/983f2213-7d03-416b-88cb-381375d587f5_characterizeImage_947140_8.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dce7cc53aca45f742a36546ead0aedf with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 5b83d199ed38296680a4da19e53808c6 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a75b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3bea1026-48ed-49b2-819b-ec6800b140c1_calibrate_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3bea1026-48ed-49b2-819b-ec6800b140c1_calibrate_947244_56.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9b525d0f-0755-4e4f-ba15-494e3219edb9_characterizeImage_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9b525d0f-0755-4e4f-ba15-494e3219edb9_characterizeImage_947244_56.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64b9ee8369cdbbb7295e4158a496d419 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 4a66821942145c731305fe207bf60735 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/30fa45d1-5082-469f-9666-b3978ffc465e_calibrate_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/30fa45d1-5082-469f-9666-b3978ffc465e_calibrate_947140_4.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/97bc108f-82c3-4c08-85f4-95f2c59abad3_characterizeImage_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/97bc108f-82c3-4c08-85f4-95f2c59abad3_characterizeImage_947140_4.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 516b9d7897df60df8765f2f2e0f425f6 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash d3a8033103bda67eb85bbdd40f622090 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/71df4647-db45-46f7-bf4c-0d31cc3de734_calibrate_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/71df4647-db45-46f7-bf4c-0d31cc3de734_calibrate_947248_18.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c1288e4e-2fec-48c9-855a-6633544f52f0_characterizeImage_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c1288e4e-2fec-48c9-855a-6633544f52f0_characterizeImage_947248_18.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b04deb76d29c718c20bac9349d16534 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 86821a3b104098cb7a5666332e1a46f2 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/af6dd874-0464-44e7-9cd4-37cbea3e76bf_characterizeImage_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/af6dd874-0464-44e7-9cd4-37cbea3e76bf_characterizeImage_947140_45.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af85ecb0569d21f6565c23f1d9d40975 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 67c1ff18f8c83a275cab436f18e84e3b parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/34d6df00-d83c-48fb-9ea8-917365a3c1aa_calibrate_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/34d6df00-d83c-48fb-9ea8-917365a3c1aa_calibrate_947144_18.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/190a557c-37ab-445a-8672-b55b02b07267_characterizeImage_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/190a557c-37ab-445a-8672-b55b02b07267_characterizeImage_947144_18.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7f94543ef50b6c4836f385a4f8c5f36 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash efd0459551c0a31b42739da3c286fb7b parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/466a90e5-6962-4084-8b75-57d9fe02a574_calibrate_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/466a90e5-6962-4084-8b75-57d9fe02a574_calibrate_947248_37.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bfdd9d8d-d5a2-447c-85f4-97296500fe72_characterizeImage_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bfdd9d8d-d5a2-447c-85f4-97296500fe72_characterizeImage_947248_37.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b70a4fb64b659932fb5efeeeaa1e30f with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 92011974b719a660bdfde311010566ca parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/950dadac-70cf-455d-8c6a-4070b8dbffeb_calibrate_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/950dadac-70cf-455d-8c6a-4070b8dbffeb_calibrate_947142_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/be35fd34-fea3-4941-a91c-c8a8dc4188d0_characterizeImage_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/be35fd34-fea3-4941-a91c-c8a8dc4188d0_characterizeImage_947142_46.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a33b7ba81e4d6691591ed047c6133e with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash c1ead47c448b7707f94c0690ab566573 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/15b0de3b-053d-4e6e-9668-3676ab9200e7_calibrate_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/15b0de3b-053d-4e6e-9668-3676ab9200e7_calibrate_947142_11.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9d360107-31f8-4e44-868c-ed63af6ebc18_characterizeImage_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9d360107-31f8-4e44-868c-ed63af6ebc18_characterizeImage_947142_11.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cba7f09399bb4e3ef1927a41e269c82c with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash c9e663974e9a0db967cea6b3403a752d parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f87f1c54-6d96-479a-a0ad-3b0e7350eb83_calibrate_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f87f1c54-6d96-479a-a0ad-3b0e7350eb83_calibrate_947142_47.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/13725f80-059f-48c4-aabc-100858ee3166_characterizeImage_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/13725f80-059f-48c4-aabc-100858ee3166_characterizeImage_947142_47.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 910563245a28e6c4f4d71f799bd14438 with result from task 85 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:34 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 9425baa0a6604febed3b3973820a1724 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cb3d5d68-435e-427b-924a-1cd9d898ec1a_calibrate_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cb3d5d68-435e-427b-924a-1cd9d898ec1a_calibrate_947248_45.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a9e887e2-3d61-4aa3-9090-15cb2ba3e1c6_characterizeImage_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a9e887e2-3d61-4aa3-9090-15cb2ba3e1c6_characterizeImage_947248_45.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881973277bdde2cb3c52d22f55f37e6a with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash ad0b4a22d4b4e316234558193def17c7 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3bde598b-f5ac-4df3-958a-4526bd10e38c_calibrate_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3bde598b-f5ac-4df3-958a-4526bd10e38c_calibrate_947142_26.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/52f890ac-bd2b-444b-b176-61e8250ddda2_characterizeImage_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/52f890ac-bd2b-444b-b176-61e8250ddda2_characterizeImage_947142_26.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 644918386555d176e43291846fbb91f2 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 3c5c9c1751c370d5963a4ee4aed420cf parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/4faaeebc-59ed-491b-921a-e1442b73f125_calibrate_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/4faaeebc-59ed-491b-921a-e1442b73f125_calibrate_947140_37.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2247ca16-ba00-44c1-896c-d64d6ae6eeb7_characterizeImage_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2247ca16-ba00-44c1-896c-d64d6ae6eeb7_characterizeImage_947140_37.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a9e9dd989bc1cdf5523a9522dffd35c with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 10fd4fa84e2670c639dbbac1844691c0 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a979c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ae5b0a24-6b5b-4bed-90b3-1219d41a9d78_calibrate_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ae5b0a24-6b5b-4bed-90b3-1219d41a9d78_calibrate_947144_6.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/93e67bda-7892-4456-9683-fe12a48155c3_characterizeImage_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/93e67bda-7892-4456-9683-fe12a48155c3_characterizeImage_947144_6.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0f01eabc45d4c8e1fa24fcccc676a36 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash a0c94d8638d400ceb27127b467986b6d parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a98e32a8-6251-410b-a8ab-d7c6b984e016_calibrate_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a98e32a8-6251-410b-a8ab-d7c6b984e016_calibrate_947244_23.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/8b550919-2929-488c-9c84-494c08a21b8f_characterizeImage_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/8b550919-2929-488c-9c84-494c08a21b8f_characterizeImage_947244_23.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33ad16bba09b4abe4db9909babc1efc6 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash e3e6670c18a3ec4ffe1dc684b1f76eb7 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fb1176c5-76ce-4726-9fde-a0d60657e385_calibrate_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fb1176c5-76ce-4726-9fde-a0d60657e385_calibrate_947140_33.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/bd5edb29-6ac5-49b7-9944-40794d421a2e_characterizeImage_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/bd5edb29-6ac5-49b7-9944-40794d421a2e_characterizeImage_947140_33.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e94f9ccd8d08126e5810cb390e93f43 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 1949662c5cc518ae9bec1aa02dfc72ec parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8bd4b43d-09f8-4b08-8b20-b01933ce7e07_calibrate_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8bd4b43d-09f8-4b08-8b20-b01933ce7e07_calibrate_947144_39.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/aa9be8d6-063c-4e83-87d8-3459594c2eab_characterizeImage_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/aa9be8d6-063c-4e83-87d8-3459594c2eab_characterizeImage_947144_39.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0860680e0c00340471b73a6c844de2d with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash d8bb004d27557a52bd376b1704f935ac parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/323177fe-06ae-4fae-a35f-2750c186f34b_calibrate_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/323177fe-06ae-4fae-a35f-2750c186f34b_calibrate_947142_9.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b137cd3e-c8af-4608-875f-4918dd2c07a8_characterizeImage_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b137cd3e-c8af-4608-875f-4918dd2c07a8_characterizeImage_947142_9.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2174fd03cf1a6891de49159be1fddfab with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash e248dba6a923af60b5b38571ef579e55 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2cbb9be2-d4db-42ad-aac6-42f67d3f1c0b_calibrate_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2cbb9be2-d4db-42ad-aac6-42f67d3f1c0b_calibrate_947142_35.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d57194a4-e535-4707-9575-c6a17c469d80_characterizeImage_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d57194a4-e535-4707-9575-c6a17c469d80_characterizeImage_947142_35.stderr parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a846362b1eeb1659df404760c67fc8f with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 3d323208259ea707f930f69a665abebe parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1af99d65-9342-4b33-b228-50d96dc74316_calibrate_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1af99d65-9342-4b33-b228-50d96dc74316_calibrate_947144_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8e79bf7c-2d20-4f13-9723-23649a3a41a6_characterizeImage_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8e79bf7c-2d20-4f13-9723-23649a3a41a6_characterizeImage_947144_31.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57f0b525918663002efc531ff1367fc7 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 97566666664a7aeba021bd909782e4a7 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d74538eb-bec4-495b-9999-0e6d4f0a7cf3_calibrate_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d74538eb-bec4-495b-9999-0e6d4f0a7cf3_calibrate_947140_50.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/ac3ca6f0-dfb0-462b-b74d-c8dd9f057d72_characterizeImage_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/ac3ca6f0-dfb0-462b-b74d-c8dd9f057d72_characterizeImage_947140_50.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bac608125216be95a73c31164f4a852 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 3deea67ca488ad6d77b809d026254442 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/397fe74d-d653-4aa8-99aa-07f5d2db2122_calibrate_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/397fe74d-d653-4aa8-99aa-07f5d2db2122_calibrate_947144_33.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0addc5f3-d625-40ab-8b6f-2a94797be963_characterizeImage_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0addc5f3-d625-40ab-8b6f-2a94797be963_characterizeImage_947144_33.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdd0b4cca9401b7bc265906f3b50835 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 2cd839efb5ccf1cd44bb356b45b5c956 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e76f64e-e25e-4937-84bd-ee73f3785faa_characterizeImage_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e76f64e-e25e-4937-84bd-ee73f3785faa_characterizeImage_947246_55.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c19930110d2674e03b167e6a8ec2eec with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 1c002eb31c67344ec0d256a689500619 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e3b5f889-5f26-4feb-8ca4-1948984c4126_calibrate_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e3b5f889-5f26-4feb-8ca4-1948984c4126_calibrate_947144_51.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/9b8d0f33-2eb3-4a66-a492-1550e40c8a60_characterizeImage_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/9b8d0f33-2eb3-4a66-a492-1550e40c8a60_characterizeImage_947144_51.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89cef9a5e13a6cfffd35e2fcb9e0814d with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 78af5e8ca6b7f3792e1ea54150e6a8f7 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/93f481ea-d893-435e-a903-a095d61bf900_calibrate_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/93f481ea-d893-435e-a903-a095d61bf900_calibrate_947144_11.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/18f14bb5-3483-45dc-baa7-e04f1fe7b3a3_characterizeImage_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/18f14bb5-3483-45dc-baa7-e04f1fe7b3a3_characterizeImage_947144_11.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9cf3c4a75d8b27e51b871d40c1d68c with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash bfa4e05a8f4217ff908c47d8f5562dd9 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/04a07320-37f5-4f9e-86f4-5adca7402f4b_calibrate_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/04a07320-37f5-4f9e-86f4-5adca7402f4b_calibrate_947246_36.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/8a59f5a3-019d-4c45-9a10-114d251b121c_characterizeImage_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/8a59f5a3-019d-4c45-9a10-114d251b121c_characterizeImage_947246_36.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a75d96a449b757a6cd3b44539ecee7b with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 6b0c7ac1b8f061015978d3a790b83267 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:35 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/ed574d67-c97f-41d8-a8ca-bcf14aba7961_calibrate_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/ed574d67-c97f-41d8-a8ca-bcf14aba7961_calibrate_947244_48.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/29384594-101d-4e4f-bb53-579c6f8428f4_characterizeImage_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/29384594-101d-4e4f-bb53-579c6f8428f4_characterizeImage_947244_48.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11bd99419a955c2b1e71dea3c6c7476a with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 4c7ebae492482b4c89974115b0e8b44d parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3bb61ef-f9a8-41cf-b4f6-fe226314f0e2_calibrate_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3bb61ef-f9a8-41cf-b4f6-fe226314f0e2_calibrate_947246_40.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/6c58e594-543c-4f5d-a8da-31eea6211585_characterizeImage_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/6c58e594-543c-4f5d-a8da-31eea6211585_characterizeImage_947246_40.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8429aa6485abf68827ea711eb9186e32 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 25b9d25c75d9907d5da371bbf2107113 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/585c66a6-4927-41df-8583-499ddc6cd4b5_calibrate_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/585c66a6-4927-41df-8583-499ddc6cd4b5_calibrate_947142_56.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/c0253d5a-03a5-4358-9012-9d91e90d64bd_characterizeImage_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/c0253d5a-03a5-4358-9012-9d91e90d64bd_characterizeImage_947142_56.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5635e758c2c796eafb4ede318e42ca71 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 268ec7ab6202715edffc4d47b53d5e76 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2f71b868-77db-47a7-993d-caaefa8e2d8c_calibrate_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2f71b868-77db-47a7-993d-caaefa8e2d8c_calibrate_947248_30.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/ee0e0f5e-8eef-4522-bbe3-30a2213f6ace_characterizeImage_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/ee0e0f5e-8eef-4522-bbe3-30a2213f6ace_characterizeImage_947248_30.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff48ff9b7e8eaede5af7bd902fdf21a with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 06c23c123f6f8f2992c7896f5eb0d728 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/b4e8a9b9-00e7-4ea4-bd7a-f16067ac80eb_calibrate_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/b4e8a9b9-00e7-4ea4-bd7a-f16067ac80eb_calibrate_947144_50.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7a77162e-f729-4e6e-bb02-ebf755b978a8_characterizeImage_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7a77162e-f729-4e6e-bb02-ebf755b978a8_characterizeImage_947144_50.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fa0a2bb959977f4c385b8c766b226c6 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 652938195e0269f9fe25c9a101453fc9 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e507c35b-c1b5-4118-8b38-114c00fee7e9_characterizeImage_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e507c35b-c1b5-4118-8b38-114c00fee7e9_characterizeImage_947244_34.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f11f2eee84ec7025f388a75640144e3 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash c92b8c1bea84ead3c13b170a74de8d53 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/215c7f2c-9548-4227-8e9c-b89df0c70bc5_calibrate_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/215c7f2c-9548-4227-8e9c-b89df0c70bc5_calibrate_947246_47.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b7d19f2e-374a-426b-b40d-dafcdbb52c20_characterizeImage_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b7d19f2e-374a-426b-b40d-dafcdbb52c20_characterizeImage_947246_47.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52d74ef9d785a4d1fdf58e32f04bb90b with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash b43755f66c32b48abafa1ab9715cf0fd parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0273b344-f050-43ac-b4e3-19d111b5ad7e_calibrate_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0273b344-f050-43ac-b4e3-19d111b5ad7e_calibrate_947142_24.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bfe9667-a9fd-43d7-8fd5-56dd8fbf71f1_characterizeImage_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bfe9667-a9fd-43d7-8fd5-56dd8fbf71f1_characterizeImage_947142_24.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e96f85582a0a556aa7b041ba026911c6 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 1f1964280b64c98a40dd1b7eba64d3ab parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/70fd144a-798f-4d2c-b86f-8d75a2944853_calibrate_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/70fd144a-798f-4d2c-b86f-8d75a2944853_calibrate_947140_6.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c575c4f8-3247-4580-b26e-6ec7319984db_characterizeImage_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c575c4f8-3247-4580-b26e-6ec7319984db_characterizeImage_947140_6.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7f4ef0ccaed14b95f83beceb0495062 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash da24c4fb20d40d57f41100edb6a4af4f parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2b045656-a550-4968-8188-2c8a28be52de_calibrate_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2b045656-a550-4968-8188-2c8a28be52de_calibrate_947142_19.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bf14c248-2c44-4477-8e26-08d0e0853a5c_characterizeImage_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bf14c248-2c44-4477-8e26-08d0e0853a5c_characterizeImage_947142_19.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde048d71d6b09e3db4030b095e2f488 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash accf7389549e22c0a73975f372bfd76e parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/412b4515-4180-4f7b-8044-eb836f50c77e_calibrate_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/412b4515-4180-4f7b-8044-eb836f50c77e_calibrate_947246_34.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fc3a07a4-665e-41a2-beb1-f4a945699549_characterizeImage_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fc3a07a4-665e-41a2-beb1-f4a945699549_characterizeImage_947246_34.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1d8718fc97ce0d8fefc42398d2ef3c with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 129c391453da4c16f9e134f03a3830a7 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/362eaf66-b47c-47cd-a915-fdeaf52f39a7_calibrate_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/362eaf66-b47c-47cd-a915-fdeaf52f39a7_calibrate_947140_25.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d8055a3-1679-46d0-8a2d-08ec78e91a58_characterizeImage_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d8055a3-1679-46d0-8a2d-08ec78e91a58_characterizeImage_947140_25.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df6de072637b22dafe640636cd1424e with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 65cf9f0b50e079f5265a3339d2cd1429 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a3e8f0b2-5402-4121-94e2-bf7189328f13_calibrate_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a3e8f0b2-5402-4121-94e2-bf7189328f13_calibrate_947144_32.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/86fd2a0e-8c12-4c59-9322-fea161faf37e_characterizeImage_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/86fd2a0e-8c12-4c59-9322-fea161faf37e_characterizeImage_947144_32.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa3b30196885dcfcd455a392763a0b7 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 94beb4c1f28eaaf7df9c1a3b9c126d96 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/593b49dc-bf9e-4b0b-9544-571fe0123eef_calibrate_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/593b49dc-bf9e-4b0b-9544-571fe0123eef_calibrate_947144_41.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3c68db00-2b4e-4879-ba2e-a32c1e060c1b_characterizeImage_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3c68db00-2b4e-4879-ba2e-a32c1e060c1b_characterizeImage_947144_41.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8a3623168894334eb3e187599afb223 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 75ea0398cea17943f91b0fd7c2564888 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5f38b036-af56-4df1-9d5e-f318d0ff8f86_calibrate_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5f38b036-af56-4df1-9d5e-f318d0ff8f86_calibrate_947244_21.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/23e5f658-d485-4e1e-bbc6-85205141d487_characterizeImage_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/23e5f658-d485-4e1e-bbc6-85205141d487_characterizeImage_947244_21.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b17430d09f15ed6e725b421b8716727 with result from task 586 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:35 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 98db006e9be6bb8987a1b53eebb208de parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5d3de7bd-44a4-478d-b049-5a0596209f2d_calibrate_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5d3de7bd-44a4-478d-b049-5a0596209f2d_calibrate_947246_46.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/682741b7-9b47-41e8-a06e-14656d01e9da_characterizeImage_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/682741b7-9b47-41e8-a06e-14656d01e9da_characterizeImage_947246_46.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 659e5c6424cc1d300f5517fab609dfde with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 0f0c07593d49df5cf4725414841555a8 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/97ca5fd3-80c4-442f-a180-aa10de889fd7_calibrate_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/97ca5fd3-80c4-442f-a180-aa10de889fd7_calibrate_947246_27.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e71cd9f7-4c8a-4cca-9ee7-ec57492314f4_characterizeImage_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e71cd9f7-4c8a-4cca-9ee7-ec57492314f4_characterizeImage_947246_27.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af2591fd8fc58827a6c9fffc8a653fc6 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash fa0ca4dd83e54c4be1da92e8543473b9 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/119aaa9a-a03b-4e21-9141-77cd8693812d_calibrate_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/119aaa9a-a03b-4e21-9141-77cd8693812d_calibrate_947144_27.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a18ae99b-1205-4485-a551-a594e7ba6cd4_characterizeImage_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a18ae99b-1205-4485-a551-a594e7ba6cd4_characterizeImage_947144_27.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7649eaff4af05cfb7b06c35bd1432a with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 4b0f30c6ff78fc006e91838b4a07adf2 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3625242b-0459-447a-95c2-740904dd2dfb_calibrate_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3625242b-0459-447a-95c2-740904dd2dfb_calibrate_947144_38.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/559e970f-72d4-48d6-b8c2-c9db35286533_characterizeImage_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/559e970f-72d4-48d6-b8c2-c9db35286533_characterizeImage_947144_38.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fabc6a8bd2e8b3b899250757c7957d8 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 31652911b1243ce3f4b84169a6356431 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ee02da3d-1662-4fde-966d-005fd936aee2_calibrate_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ee02da3d-1662-4fde-966d-005fd936aee2_calibrate_947246_21.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e9d05d97-4b4d-4758-b3ef-785ca963922e_characterizeImage_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/e9d05d97-4b4d-4758-b3ef-785ca963922e_characterizeImage_947246_21.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0776ea8efa22d9769fa7cf8d5058a8aa with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 739878c69c522942b83067366216c146 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c1fc705f-c0e7-4a76-881f-ff61592cda51_calibrate_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c1fc705f-c0e7-4a76-881f-ff61592cda51_calibrate_947140_62.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cb451032-49f6-458f-96ed-b7b3b5a96e86_characterizeImage_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cb451032-49f6-458f-96ed-b7b3b5a96e86_characterizeImage_947140_62.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e257d7ac1b47da51ffd7f57ff9e216e0 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash dd3eea06582399248fcd3c2872601832 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/06b63ba7-f9e0-4e91-981e-203542840dcd_calibrate_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/06b63ba7-f9e0-4e91-981e-203542840dcd_calibrate_947246_3.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f0886063-00dc-4afa-ad65-cc9f7f22b608_characterizeImage_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f0886063-00dc-4afa-ad65-cc9f7f22b608_characterizeImage_947246_3.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baf95c2cc6f155dd2c3be4f0eecb2397 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 0eb9d9ef6ca1c8dbeec8989829b9030b parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/05c7c642-fdc2-476c-af84-95082cb07f80_calibrate_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/05c7c642-fdc2-476c-af84-95082cb07f80_calibrate_947144_28.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fc50836a-0bcd-47e2-b17c-f92fa7ff632d_characterizeImage_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fc50836a-0bcd-47e2-b17c-f92fa7ff632d_characterizeImage_947144_28.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f82d6542b14d4c5ea8f43f51454bceb with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash ff473303940088f21621b682d4e2b184 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b987c074-cc20-4799-9a03-64680f967eed_calibrate_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b987c074-cc20-4799-9a03-64680f967eed_calibrate_947142_8.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d6adb8c5-edaa-493a-b014-809d6922e788_characterizeImage_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d6adb8c5-edaa-493a-b014-809d6922e788_characterizeImage_947142_8.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7746d9e26baf4d531fcb8d45472ba8cb with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash d24e32a13609c91ba99d7d4b5ff687f6 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/456e2735-9986-44c2-b997-b8dfaa32c48f_calibrate_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/456e2735-9986-44c2-b997-b8dfaa32c48f_calibrate_947144_24.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/04bfe95b-2beb-4658-9869-60e949ab4f75_characterizeImage_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/04bfe95b-2beb-4658-9869-60e949ab4f75_characterizeImage_947144_24.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24f9c01fb7174e2183bee030c732900c with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash fa3a3f5ea3a95ff33572fa42c6d67016 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/046ccdf4-8273-44f8-8f6e-517a909fa0d0_calibrate_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/046ccdf4-8273-44f8-8f6e-517a909fa0d0_calibrate_947248_26.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f36d4638-fea1-472d-bcf4-79f20b4c0661_characterizeImage_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f36d4638-fea1-472d-bcf4-79f20b4c0661_characterizeImage_947248_26.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e55fe97fd85a6a5221bed37ccfa7d91 with result from task 307 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:35 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 15e78ec67444cd32bc1772192c7a5746 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:36 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3e53fd32-97c9-42bf-9a1c-321629409d34_calibrate_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3e53fd32-97c9-42bf-9a1c-321629409d34_calibrate_947246_39.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d04ca1a2-eee2-48ae-a9dc-2d450624fd62_characterizeImage_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d04ca1a2-eee2-48ae-a9dc-2d450624fd62_characterizeImage_947246_39.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1acfe2efce405460bc7d7153010d8a with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 16b3f9a70acc6983f0dec054d601a9bd parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/20ad85c2-4949-4197-8472-13312739b264_calibrate_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/20ad85c2-4949-4197-8472-13312739b264_calibrate_947244_35.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0286c90f-d9ae-49d7-9c53-09461a5dbf25_characterizeImage_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0286c90f-d9ae-49d7-9c53-09461a5dbf25_characterizeImage_947244_35.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3adf52d0e8a53e316bb44fa78037263 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 01cb633c003fa3efbf97c3effd72b2dc parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:39 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/447dcf36-786f-497f-a0fe-740b7ba06c64_calibrate_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/447dcf36-786f-497f-a0fe-740b7ba06c64_calibrate_947142_55.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/492b2a4c-afb0-47d7-b986-c594ba94cc1f_characterizeImage_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/492b2a4c-afb0-47d7-b986-c594ba94cc1f_characterizeImage_947142_55.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee7720abca20f442556dae4bb2fce668 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash d1b30dceb1f78e6349764c815aaf7f00 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/77cd0ca7-c0cd-4890-b2fe-4293ea9f1d9f_calibrate_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/77cd0ca7-c0cd-4890-b2fe-4293ea9f1d9f_calibrate_947248_21.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/163018e0-3865-4b8f-b779-49b25c9f8f59_characterizeImage_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/163018e0-3865-4b8f-b779-49b25c9f8f59_characterizeImage_947248_21.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f2d42e249424cd560b13fe3f83f2f3f with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash aae658d3fcbfcdf6105d772f4a007af1 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cebd914a-6bf1-4c05-a1b6-94f1f3a8e364_calibrate_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cebd914a-6bf1-4c05-a1b6-94f1f3a8e364_calibrate_947142_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/38a8ec16-be99-41b8-ada0-e2efd5f1accd_characterizeImage_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/38a8ec16-be99-41b8-ada0-e2efd5f1accd_characterizeImage_947142_31.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51fac95490741903c70908c93e94f0c7 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 5036a03ceaa4e4e6ed530d67e4439640 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6287d4e1-46c5-472d-9b88-7a4c475f5d84_calibrate_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6287d4e1-46c5-472d-9b88-7a4c475f5d84_calibrate_947246_35.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/07699eb7-a52d-48c5-92c3-9a5da11e0ba2_characterizeImage_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/07699eb7-a52d-48c5-92c3-9a5da11e0ba2_characterizeImage_947246_35.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a35ad8c391b86b853392d32b8481aa10 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 89f16adf3b1b3ba37dfbf35d63cd39e5 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/dbfbcf17-ef52-4329-b9b7-37bc8c39344f_calibrate_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/dbfbcf17-ef52-4329-b9b7-37bc8c39344f_calibrate_947246_10.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/3e8e8eb3-bd10-442e-a24a-79ad932445c5_characterizeImage_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/3e8e8eb3-bd10-442e-a24a-79ad932445c5_characterizeImage_947246_10.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab929729c1f5fd6392093a47e9c6d579 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash ac09530b4e37b672a12fc994f5237ecb parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0546d703-380e-498e-b4f6-505f45b0630f_characterizeImage_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0546d703-380e-498e-b4f6-505f45b0630f_characterizeImage_947248_59.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f54e6e9b4f5995b782e3a2e08ec45a47 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 95466560bcf9dcdbc1df3079f8a2e7ff parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:39 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6276ca36-ca07-4252-97ee-c5d7ffcc80ad_calibrate_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6276ca36-ca07-4252-97ee-c5d7ffcc80ad_calibrate_947246_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9b698916-78d2-42b6-a4c6-cc5d6f2831c5_characterizeImage_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9b698916-78d2-42b6-a4c6-cc5d6f2831c5_characterizeImage_947246_22.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 764e347edfb55f3690428ed720425b55 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash a9adccf057b3f7f08a2bf9f3d71003bd parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a57dd677-e3aa-4092-b6df-1cd883244cbd_calibrate_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a57dd677-e3aa-4092-b6df-1cd883244cbd_calibrate_947244_39.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/582c43cb-357a-4b08-8fe4-8d53a5be365d_characterizeImage_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/582c43cb-357a-4b08-8fe4-8d53a5be365d_characterizeImage_947244_39.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c673e76e9b928eb0c6cd5ddba7776b60 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 6ed78ab5fa4691791f63e78d07de7a78 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/77808938-4245-42a6-9a06-b5056d11799b_calibrate_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/77808938-4245-42a6-9a06-b5056d11799b_calibrate_947244_45.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/430b20cd-6033-4571-b530-236582da9c1a_characterizeImage_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/430b20cd-6033-4571-b530-236582da9c1a_characterizeImage_947244_45.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30530524abb1d72baabcce1c80542868 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 3ee76e12d233d5011ccad13d4fa94d3a parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/75be186b-b81e-45a7-8ad8-96411d2d595a_calibrate_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/75be186b-b81e-45a7-8ad8-96411d2d595a_calibrate_947140_19.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2b6c577e-bcb0-4d3f-bfec-6691a3ad0d3d_characterizeImage_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2b6c577e-bcb0-4d3f-bfec-6691a3ad0d3d_characterizeImage_947140_19.stderr parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91a714fac7894bc7a06db902087cd58 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 8796730398f8047b67e672d623409f70 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/230582ab-3291-4b72-a902-8edeb65950c8_calibrate_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/230582ab-3291-4b72-a902-8edeb65950c8_calibrate_947144_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a7ea58cf-9bd0-40d0-8fcf-17bbacabda1d_characterizeImage_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a7ea58cf-9bd0-40d0-8fcf-17bbacabda1d_characterizeImage_947144_53.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 481cd287556e6a58b8c5fad9b4217aca with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 52f7948506536ea1c0daa1f6b65facb3 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:40 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/381cee70-3684-4720-9878-72a3fb19448b_calibrate_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/381cee70-3684-4720-9878-72a3fb19448b_calibrate_947244_38.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9773dc17-b175-4fbf-931f-bc0dca49fa09_characterizeImage_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9773dc17-b175-4fbf-931f-bc0dca49fa09_characterizeImage_947244_38.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d2dbf8e3444c4500d09611825872c1 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 72b6ce15c67f8f17e714553fa206b39f parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7eb33e41-eb69-4777-81a8-9fd67999a4f5_calibrate_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7eb33e41-eb69-4777-81a8-9fd67999a4f5_calibrate_947142_62.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/71215b4d-9ef8-4e36-8863-344e0406da84_characterizeImage_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/71215b4d-9ef8-4e36-8863-344e0406da84_characterizeImage_947142_62.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94d95da50032a68eaacddf2a7d52ad2e with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 891f8ee35997c268b06611b74102e416 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3450d34a-2e6b-4c18-8fe1-fef612fddec6_calibrate_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3450d34a-2e6b-4c18-8fe1-fef612fddec6_calibrate_947144_16.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1cd1a098-e030-4136-88b1-d8263a65ee4f_characterizeImage_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1cd1a098-e030-4136-88b1-d8263a65ee4f_characterizeImage_947144_16.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6d541015d6477319f526bb7ad29c9a3 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash fbbf13947059badee0825443842af258 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/e49cce65-8a12-4207-ab92-33918e518787_characterizeImage_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/e49cce65-8a12-4207-ab92-33918e518787_characterizeImage_947248_58.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d3e178f9b16066c14c8f8924608a921 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 3c8e4edfe83177e3725ba6367fd9693e parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c25c04b4-47d6-40b0-ad96-217f9f0e0384_characterizeImage_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c25c04b4-47d6-40b0-ad96-217f9f0e0384_characterizeImage_947248_40.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8363d7d28d21fb4af93edb6132f63ddf with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash c28c8fd81c2496ce0c3167f05dde5cfe parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c198debe-7e65-42a2-9a80-ea033ee5f57f_calibrate_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c198debe-7e65-42a2-9a80-ea033ee5f57f_calibrate_947244_10.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/d6b5b9e1-f256-4b2b-9ec1-79e3d0d6dae0_characterizeImage_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/d6b5b9e1-f256-4b2b-9ec1-79e3d0d6dae0_characterizeImage_947244_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 772 try 0 failed parsl.dataflow.dflow INFO: Task 772 marked for retry parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 17108cb988fa9bfbc8816f60b7972700 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 1 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc86d3a65cef6383a6bad7218fd997a1 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 87168c802d095579cf40dae5556994e9 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b3d64c08-33c9-41db-bc8b-b92ae4c59843_calibrate_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b3d64c08-33c9-41db-bc8b-b92ae4c59843_calibrate_947248_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a3760520-476b-42cb-b17e-54d3e5a76047_characterizeImage_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a3760520-476b-42cb-b17e-54d3e5a76047_characterizeImage_947248_39.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 477780e0bb56ea66eadd2162021152ea with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash aa1c910a05d5678cb334310dd374ac6c parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c9353a11-d125-4d83-a07b-181d988653a9_calibrate_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c9353a11-d125-4d83-a07b-181d988653a9_calibrate_947140_31.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d2c3c97-7fc1-4eae-9524-8952b7aac9f8_characterizeImage_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/2d2c3c97-7fc1-4eae-9524-8952b7aac9f8_characterizeImage_947140_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 1 failed parsl.dataflow.dflow ERROR: Task 214 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bf99c1e847d3fccc5fb91e6f18040f61 with result from task 214 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 215 try 0 failed parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1d0b5b84-bff0-47ad-afd9-2c606701d25e_calibrate_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1d0b5b84-bff0-47ad-afd9-2c606701d25e_calibrate_947244_2.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/4ab2ed26-5396-46a5-a5c4-143c1b84520a_characterizeImage_947244_2.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3fd21ea6f22e8101cf627b0fb7b795b with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash d61ff28bff6e7e7f30af81575b4ad18c parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/62501877-176a-4017-9528-ecc7dc9737f9_calibrate_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/62501877-176a-4017-9528-ecc7dc9737f9_calibrate_947248_14.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7d4c6ed6-451f-4fc4-82bf-0b2a40976805_characterizeImage_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7d4c6ed6-451f-4fc4-82bf-0b2a40976805_characterizeImage_947248_14.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb5cb0f9b612cbbc7bd14b3bf29b527e with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 4ea61d2aa49fdb4fc8694472cb4677d6 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5c4e5e78-b626-4c60-a9ec-3fb44d7f9e2a_calibrate_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5c4e5e78-b626-4c60-a9ec-3fb44d7f9e2a_calibrate_947246_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5de5eb78-cc3a-4a53-9809-4a06c9a5de0c_characterizeImage_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5de5eb78-cc3a-4a53-9809-4a06c9a5de0c_characterizeImage_947246_9.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6537c95e96476300deebfbb8b7b9fd1d with result from task 472 parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:40 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 3c135b7d9ae6f4fc8d76e6b3415d630c parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/deadd269-99ed-4817-9a8d-9df540328014_calibrate_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/deadd269-99ed-4817-9a8d-9df540328014_calibrate_947246_13.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c64f2059-9c1c-4cf0-82e5-50d58b36f72f_characterizeImage_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c64f2059-9c1c-4cf0-82e5-50d58b36f72f_characterizeImage_947246_13.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f90fc6c73dd74cf36a72d3709c012fec with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash dbe0807f20c154dc4272039e52d091bc parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/441ce415-7fb6-4e81-80b5-523990766793_calibrate_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/441ce415-7fb6-4e81-80b5-523990766793_calibrate_947142_37.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a843efa3-956c-4212-aa48-0d2318562846_characterizeImage_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a843efa3-956c-4212-aa48-0d2318562846_characterizeImage_947142_37.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a436c7581e8283588e7e9cc4fd7a7d2 with result from task 1081 parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:40 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash d9354935f40825340216b3619580c496 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d1fd6468-2b11-4874-bb0d-ce5f77b8dbed_calibrate_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d1fd6468-2b11-4874-bb0d-ce5f77b8dbed_calibrate_947246_14.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/da0829b7-616d-4c79-aa59-7a0b1e4fc079_characterizeImage_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/da0829b7-616d-4c79-aa59-7a0b1e4fc079_characterizeImage_947246_14.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9be903b5838c422b91a1dfa4d2a90e3 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash b284d56e5949d9b9f8a924f342cb327f parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7033ce03-0969-4f6e-be1d-f576cb6a975c_calibrate_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7033ce03-0969-4f6e-be1d-f576cb6a975c_calibrate_947246_56.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fdb7ec32-dac2-419a-bf47-e804a03f6607_characterizeImage_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fdb7ec32-dac2-419a-bf47-e804a03f6607_characterizeImage_947246_56.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec785c74eb132ee40046ba894b2b1a9c with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 7b95237544f06b72e2e0d38cebd4a487 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:40 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/775f8c7d-b49d-4073-93bd-ccf3d5df272b_calibrate_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/775f8c7d-b49d-4073-93bd-ccf3d5df272b_calibrate_947246_30.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/913ff464-fbb5-4e9e-af51-e3905ef86e40_characterizeImage_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/913ff464-fbb5-4e9e-af51-e3905ef86e40_characterizeImage_947246_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 322 try 0 failed parsl.dataflow.dflow INFO: Task 322 marked for retry parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash c9ab4c4ddd0cd5ff9e2e33d5fc726a0a parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 1 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aabc2e534c7d7d0738f62ace1b3c41f with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash ed795ccebc4601e73a6e1c03d6d6222b parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1cbcf670-41ce-441e-ba4a-4d2b64053e7a_calibrate_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1cbcf670-41ce-441e-ba4a-4d2b64053e7a_calibrate_947144_25.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/29862f05-0b28-4d84-ad0f-fca665c39d99_characterizeImage_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/29862f05-0b28-4d84-ad0f-fca665c39d99_characterizeImage_947144_25.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df7dab5c8c4f635320218215cfcb0e51 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash f7e3b9daa0606fc0c23b52219169ad30 parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/78d38a3c-3246-4c0a-85ae-5e06f2023c60_calibrate_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/78d38a3c-3246-4c0a-85ae-5e06f2023c60_calibrate_947244_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/932f2fe0-c54e-4826-adf9-463987f07737_characterizeImage_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/932f2fe0-c54e-4826-adf9-463987f07737_characterizeImage_947244_37.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80ffb2c5705a2c779ef69d6482b3eac1 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 991b4419b35107ac3ae90d80ec10c867 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/e54082fe-c4ec-4e4d-b3a8-cb5cc1d4c455_calibrate_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/e54082fe-c4ec-4e4d-b3a8-cb5cc1d4c455_calibrate_947246_6.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/41afc57b-552b-40a3-9f98-21e6dfc8bc6f_characterizeImage_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/41afc57b-552b-40a3-9f98-21e6dfc8bc6f_characterizeImage_947246_6.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d9c93bd066ea3398e566a49a119b8e2 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash b6937edff8b0e9bedb957df20b909dfb parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bf39de38-cec8-4129-b864-8d16601903ae_calibrate_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bf39de38-cec8-4129-b864-8d16601903ae_calibrate_947248_55.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/1624d715-3ba0-4b45-8002-e4ed46d0da7b_characterizeImage_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/1624d715-3ba0-4b45-8002-e4ed46d0da7b_characterizeImage_947248_55.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478b3856fe948d82a89cb5becaf79e9e with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash e274537a700cdcfa4fc1e3a9d3b5345c parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1c54eb48-e2af-4278-ac44-0e0182f16fe9_calibrate_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1c54eb48-e2af-4278-ac44-0e0182f16fe9_calibrate_947244_47.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ba2578c3-f50e-4889-bc7f-cb9c1df64067_characterizeImage_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ba2578c3-f50e-4889-bc7f-cb9c1df64067_characterizeImage_947244_47.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3164362c31b290b07b46637a855e36f0 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 460ca9782f903957208fc0f9c7544a87 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a76de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:41 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3dab620-1d03-4cbf-be5f-d9249ebd6bff_calibrate_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3dab620-1d03-4cbf-be5f-d9249ebd6bff_calibrate_947246_15.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/74757817-24a1-451e-a321-b71e8ed8c2a2_characterizeImage_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/74757817-24a1-451e-a321-b71e8ed8c2a2_characterizeImage_947246_15.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0549628f9b84ad4f92a29a465943e8d with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 3dea2fbd18d28ce834da06aa67b8fb09 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/056fb217-1440-49b7-a83e-5e942f2bd4ae_calibrate_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/056fb217-1440-49b7-a83e-5e942f2bd4ae_calibrate_947246_33.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/28596b8a-446a-42de-ab51-24faeef78703_characterizeImage_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/28596b8a-446a-42de-ab51-24faeef78703_characterizeImage_947246_33.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec364434290312ede7f3aabb627fa4a1 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash bffcf29fcca3ab7323d6db0d0e627d90 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d385b13d-510e-4279-b1e3-c838b2567ef2_calibrate_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d385b13d-510e-4279-b1e3-c838b2567ef2_calibrate_947142_45.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/447bdd8c-3efa-4525-b5ea-7041a95ec749_characterizeImage_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/447bdd8c-3efa-4525-b5ea-7041a95ec749_characterizeImage_947142_45.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b8227d197b73e5248064cfe118482a3 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 92c03e0b417a58fec98f7344284807cd parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a9ab60fe-1833-4007-b2b4-b69f16ea0900_calibrate_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a9ab60fe-1833-4007-b2b4-b69f16ea0900_calibrate_947246_62.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9949c787-83f8-4ed8-b0b3-53cf7fd19226_characterizeImage_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/9949c787-83f8-4ed8-b0b3-53cf7fd19226_characterizeImage_947246_62.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e96a1f72fab3ef854b9f966a7d557af3 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash e4c77fa581753ba50bc3d0733dad0501 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e96e251b-425c-4e33-988a-aa8ba011aaa0_calibrate_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e96e251b-425c-4e33-988a-aa8ba011aaa0_calibrate_947144_49.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/91a9f55d-1562-4aa3-89cd-7e84d349b732_characterizeImage_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/91a9f55d-1562-4aa3-89cd-7e84d349b732_characterizeImage_947144_49.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ae1c5b42d975a5188636dc2386ed91 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 69d3ce363ae17e4de6e1e6abf7b297c9 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/860f7316-1205-4a42-8802-1cbf199382ae_calibrate_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/860f7316-1205-4a42-8802-1cbf199382ae_calibrate_947142_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/339282ea-4fdb-46ca-8abf-31ffa5bfc015_characterizeImage_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/339282ea-4fdb-46ca-8abf-31ffa5bfc015_characterizeImage_947142_30.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e1ddc96156727eedd8a09f5c4927d36 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash b97dae25bbac4c0de8472f9c1411926e parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b30d28b0-8837-473e-bd4f-a91976e588e7_calibrate_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b30d28b0-8837-473e-bd4f-a91976e588e7_calibrate_947244_6.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/000528ed-8c82-419c-9d00-ab5d0adacf15_characterizeImage_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/000528ed-8c82-419c-9d00-ab5d0adacf15_characterizeImage_947244_6.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85bc2faf2cf616edfff57c0e7042f1b4 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 4303df5a728b5e45953c4df032072df9 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a5d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2d851ad2-2102-4bc1-9ac9-6ccf02eafac4_calibrate_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2d851ad2-2102-4bc1-9ac9-6ccf02eafac4_calibrate_947246_44.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/821cf2a1-23f1-4e35-9d66-7919b5ee9490_characterizeImage_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/821cf2a1-23f1-4e35-9d66-7919b5ee9490_characterizeImage_947246_44.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bf64fe0fb13b3665ee1fd8cd1dd5ba9 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 039076019bd6e36828def09aad960f10 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:46 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/677d4273-7fbb-4dc6-a531-518008222cc0_calibrate_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/677d4273-7fbb-4dc6-a531-518008222cc0_calibrate_947248_15.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4875527b-c883-412b-98d6-db04a8d6f58b_characterizeImage_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4875527b-c883-412b-98d6-db04a8d6f58b_characterizeImage_947248_15.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f66dccbdb745fd5472e8999bb2c88a8b with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 718bd08871d0f9d3d8edac478616ea9d parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d606e417-3269-48da-a33d-b94501c06748_calibrate_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d606e417-3269-48da-a33d-b94501c06748_calibrate_947246_58.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f02a3f08-b30d-4649-816c-988d2656a830_characterizeImage_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f02a3f08-b30d-4649-816c-988d2656a830_characterizeImage_947246_58.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d395992b81eaed01272177dd7dce81e3 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 1ae2efff0380516fa6c5fcbc6d7534fe parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:49 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:50 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/9bebec98-c87d-422e-b06f-b00293d4eb0b_calibrate_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/9bebec98-c87d-422e-b06f-b00293d4eb0b_calibrate_947244_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/576c0279-80a9-4236-b566-964df7b6e51d_characterizeImage_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/576c0279-80a9-4236-b566-964df7b6e51d_characterizeImage_947244_8.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f0e470a102f26c1706dc024d0906fad with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 4691fd950c9e08f5636121904f416c52 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/120dbe16-6fd5-41a0-83b6-9186f0f7d8c4_calibrate_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/120dbe16-6fd5-41a0-83b6-9186f0f7d8c4_calibrate_947246_8.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f5f8bdf8-9a28-4352-8c1e-3293441068bc_characterizeImage_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f5f8bdf8-9a28-4352-8c1e-3293441068bc_characterizeImage_947246_8.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514193de9a982ad2bc5bd7fb42fc9297 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash f82434364db615261dc8f4d36310dba0 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:50 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b7250007-43e3-45c5-adc8-9ba9692d7e03_calibrate_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b7250007-43e3-45c5-adc8-9ba9692d7e03_calibrate_947246_7.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5496ad54-a1e3-45ce-b98a-b03d6b41bb0d_characterizeImage_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5496ad54-a1e3-45ce-b98a-b03d6b41bb0d_characterizeImage_947246_7.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f7778b06966815189f23f29a33b7fb with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 2363cfcecdefaf3436b46bbb97c2efa5 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:52 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c4ff16c2-7f67-4999-bc39-98e81eb01ad3_calibrate_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c4ff16c2-7f67-4999-bc39-98e81eb01ad3_calibrate_947246_52.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5cd9714c-395f-400b-975d-fa28832ca61f_characterizeImage_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5cd9714c-395f-400b-975d-fa28832ca61f_characterizeImage_947246_52.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14a54e457700d67da95927b3d8ab9021 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 8f2dc5c0f01f74bc9c8110e717ab66e3 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a95d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:54 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03dc62b8-a844-4b42-a19e-28439403f7d6_calibrate_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03dc62b8-a844-4b42-a19e-28439403f7d6_calibrate_947248_1.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cc4a68a2-84c8-4a9f-b7d4-fbbcba82071f_characterizeImage_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cc4a68a2-84c8-4a9f-b7d4-fbbcba82071f_characterizeImage_947248_1.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 263470a00f515e679ac23b3b6a19ae72 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 9d3874dd1e84935c0edee623e521bcfc parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03705dfa-a993-4cfe-9ace-0be4b14588f5_calibrate_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03705dfa-a993-4cfe-9ace-0be4b14588f5_calibrate_947248_43.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0edd00a7-4415-436a-85ec-599eb719157e_characterizeImage_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0edd00a7-4415-436a-85ec-599eb719157e_characterizeImage_947248_43.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac5b0569b33b3d76e54c6524bb25e65 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash e517f4910dd4f4fa8c9c042e781f3d81 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:36:55 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/582412ca-6feb-4590-8d46-2d3a53fd1b58_calibrate_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/582412ca-6feb-4590-8d46-2d3a53fd1b58_calibrate_947140_53.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/01b32ded-fc56-409b-ac32-b83184ca5db6_characterizeImage_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/01b32ded-fc56-409b-ac32-b83184ca5db6_characterizeImage_947140_53.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff0e6b11bb6a82043aca36077b275a96 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 68fe2ac6ddb7a4de26ceb772dcf64e70 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/63551d04-db9e-4784-9a26-2aea4d0499f8_calibrate_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/63551d04-db9e-4784-9a26-2aea4d0499f8_calibrate_947140_1.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/fdeb31eb-c3b6-4cd0-846a-17f602f50194_characterizeImage_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/fdeb31eb-c3b6-4cd0-846a-17f602f50194_characterizeImage_947140_1.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5d50f699bfa41f62192eb0dd327e2ce with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash e221758c15b6cb259a9e39d2fa6b5125 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:00 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:00 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6510f008-6292-4a92-b0c0-7eabed6ab407_calibrate_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6510f008-6292-4a92-b0c0-7eabed6ab407_calibrate_947246_5.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fd0c7180-38a1-42a4-9685-a150136f08fb_characterizeImage_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/fd0c7180-38a1-42a4-9685-a150136f08fb_characterizeImage_947246_5.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37da7a3bbc980109d2159e466fa87010 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 9a80c739cfd5a44c88052b9416627ab2 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4b088098-a2d0-4f43-8708-1bff75550091_calibrate_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4b088098-a2d0-4f43-8708-1bff75550091_calibrate_947144_40.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d414efcd-7dc9-4174-8980-52150c97f168_characterizeImage_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d414efcd-7dc9-4174-8980-52150c97f168_characterizeImage_947144_40.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c9b8742cde94db522b458446ad60a01 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 0c8bc26261cdfe812ac32c5e3ef9f2bd parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/462639d4-0e2d-4267-94f9-f4892cbacf65_calibrate_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/462639d4-0e2d-4267-94f9-f4892cbacf65_calibrate_947142_23.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5214ea49-38f0-40bc-89db-32c95d28969a_characterizeImage_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5214ea49-38f0-40bc-89db-32c95d28969a_characterizeImage_947142_23.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3f93601d47b573e6e91d0ccff86a76 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash d562f55345650655814e6eedd50ccd9b parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:00 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/85bd69cf-06ed-4917-a30e-859a67e3ba3e_calibrate_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/85bd69cf-06ed-4917-a30e-859a67e3ba3e_calibrate_947246_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0fc72a20-1653-4dec-a335-319c50cd598b_characterizeImage_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0fc72a20-1653-4dec-a335-319c50cd598b_characterizeImage_947246_16.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66e9b434eae81a8f4812f8dfd6a48c91 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 90b7b17256ae31b488a88e3f9b7fcd81 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:01 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/f43147d1-e611-4aae-af08-5188369f8414_calibrate_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/f43147d1-e611-4aae-af08-5188369f8414_calibrate_947140_7.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e14f66a9-bf4a-41db-9993-385a321fa488_characterizeImage_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e14f66a9-bf4a-41db-9993-385a321fa488_characterizeImage_947140_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 571 try 1 failed parsl.dataflow.dflow ERROR: Task 571 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ffdde4c0e7cc88efec14383db4adddc4 with result from task 571 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 572 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 572 try 0 failed parsl.dataflow.dflow INFO: Task 572 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/88f4f51d-4a49-4594-8f12-a045e7a9c5bb_calibrate_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/88f4f51d-4a49-4594-8f12-a045e7a9c5bb_calibrate_947246_2.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2e3ca483-0d63-44b9-b6ee-e1f3239a1019_characterizeImage_947246_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 928 try 0 failed parsl.dataflow.dflow INFO: Task 928 marked for retry parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash f90cfdff87e00ff246e68340692c541e parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 1 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6af9ab30c4b07d67eb9a0027558b01 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 263f0c123edddaddbfa7094945998a5d parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/539f94ee-136d-43f4-8acb-9f4e98d13b45_calibrate_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/539f94ee-136d-43f4-8acb-9f4e98d13b45_calibrate_947248_44.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5dcce05a-dace-419e-ae9b-3934a548f416_characterizeImage_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/5dcce05a-dace-419e-ae9b-3934a548f416_characterizeImage_947248_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1096 try 1 failed parsl.dataflow.dflow ERROR: Task 1096 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1a7050a4d731f912cb8615601fcee24f with result from task 1096 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 1097 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 1097 try 0 failed parsl.dataflow.dflow INFO: Task 1097 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3a2b27f2-cce6-4722-b7a8-1ad11b100798_calibrate_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3a2b27f2-cce6-4722-b7a8-1ad11b100798_calibrate_947140_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e184d51e-2389-4ec7-8b23-7522d1df0c32_characterizeImage_947140_2.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7eadaa9017b7dfe3b71493d77adfad with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 5a60bbf082222cc9c6395e75e4ab26cb parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b24da02c-ca23-41ea-81f5-3dfd9db7a9ff_calibrate_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b24da02c-ca23-41ea-81f5-3dfd9db7a9ff_calibrate_947246_19.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a706c7c4-b726-458d-98ec-4bef47b60ed5_characterizeImage_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a706c7c4-b726-458d-98ec-4bef47b60ed5_characterizeImage_947246_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 169 try 1 failed parsl.dataflow.dflow ERROR: Task 169 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 15ecd42793d43aaa5e82231680c692e2 with result from task 169 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:03 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 170 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 170 try 0 failed parsl.dataflow.dflow INFO: Task 170 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7956ef14-ee2f-425a-bbc1-8ec3cd274d32_calibrate_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7956ef14-ee2f-425a-bbc1-8ec3cd274d32_calibrate_947248_2.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a0e5cecf-0d3f-4ff1-8879-78695ddece8c_characterizeImage_947248_2.stderr parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ce470fa77f45dd925a2631cc8ee895 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash f3d93a7362526b434c5ad841760d2786 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/49bbb636-5b25-4171-b11b-3cbe78ff1bd7_calibrate_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/49bbb636-5b25-4171-b11b-3cbe78ff1bd7_calibrate_947244_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/27d47967-c3cf-4256-8970-7188f57e351a_characterizeImage_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/27d47967-c3cf-4256-8970-7188f57e351a_characterizeImage_947244_53.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b211247b6289953e5beb393a03d69cf with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 7e887ddb4f423ceb190d59846d9a0d32 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7af8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/57966154-127f-45da-b757-88ebeba118fa_calibrate_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/57966154-127f-45da-b757-88ebeba118fa_calibrate_947244_24.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/bdc87192-98c0-4861-b886-d1e156948ab4_characterizeImage_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/bdc87192-98c0-4861-b886-d1e156948ab4_characterizeImage_947244_24.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4b6e364ae736f78db86491c9c268cc7 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 1ad2e19ba24a793940bff4a2cbecf346 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cee8d462-ee85-4680-b18f-d14d3473d83f_calibrate_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cee8d462-ee85-4680-b18f-d14d3473d83f_calibrate_947142_6.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/241e08df-7a70-4a4b-9dd0-be207542dba3_characterizeImage_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/241e08df-7a70-4a4b-9dd0-be207542dba3_characterizeImage_947142_6.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667291a4d330d2c9e21660dc4fbb8d8c with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 125aae77f9c007d9b79a247e4d88b0bf parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ae56a859-391b-4805-a17e-c2c99f141e09_calibrate_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ae56a859-391b-4805-a17e-c2c99f141e09_calibrate_947140_60.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d7e82b5e-222a-43a5-83ab-116c7d9e0f73_characterizeImage_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d7e82b5e-222a-43a5-83ab-116c7d9e0f73_characterizeImage_947140_60.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf015058304821bce5eb6901a56319f with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 5ea2cd8cbe33c3b3c4e994c3ce413110 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f7dc9fb6-5bce-4613-ac1e-f0495dc64655_calibrate_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f7dc9fb6-5bce-4613-ac1e-f0495dc64655_calibrate_947144_19.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/25edbba0-a670-47c5-a7a3-4e9a3940d7c4_characterizeImage_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/25edbba0-a670-47c5-a7a3-4e9a3940d7c4_characterizeImage_947144_19.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe37a00a40cddfd3a3d0b92cf191b78 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 7230a1fc8b00d3c9cda5588895b285a1 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7115362f-5706-4b03-8802-63751fb03b65_calibrate_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7115362f-5706-4b03-8802-63751fb03b65_calibrate_947248_3.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0cef5826-58fc-480d-a803-1cd82693d351_characterizeImage_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0cef5826-58fc-480d-a803-1cd82693d351_characterizeImage_947248_3.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45fcea161a1cb4ed61a3e53686612444 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 085787e6977daeef94283eb2e1bcf09e parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f986bc2a-33be-4d14-b178-16f8f240ca71_calibrate_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f986bc2a-33be-4d14-b178-16f8f240ca71_calibrate_947248_53.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2fcbfb45-d388-40d4-8b75-8a361428e207_characterizeImage_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2fcbfb45-d388-40d4-8b75-8a361428e207_characterizeImage_947248_53.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 832542642cc7c0684c170f0f05e398ed with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 1a07adc291938ecee21fc7dee7984d26 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/97a229e8-736b-4831-9be7-5be60fd51d6f_calibrate_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/97a229e8-736b-4831-9be7-5be60fd51d6f_calibrate_947248_49.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/193b5159-6112-450c-949d-a8a5c73f0a58_characterizeImage_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/193b5159-6112-450c-949d-a8a5c73f0a58_characterizeImage_947248_49.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60c45864565b28a2e258bf04f0a5f4a1 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash b01f6b1ea1a079b0c4b35865f824ef96 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5d7553dc-5528-4f3d-8668-91e36d07f8b7_calibrate_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5d7553dc-5528-4f3d-8668-91e36d07f8b7_calibrate_947142_1.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bf652a6-6df6-4269-9ab6-13d597dd0c41_characterizeImage_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5bf652a6-6df6-4269-9ab6-13d597dd0c41_characterizeImage_947142_1.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 322 try 1 failed parsl.dataflow.dflow ERROR: Task 322 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c9ab4c4ddd0cd5ff9e2e33d5fc726a0a with result from task 322 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:03 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 323 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 323 try 0 failed parsl.dataflow.dflow INFO: Task 323 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3076a401-2c4c-4c61-880b-d0f5711fd3b6_calibrate_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3076a401-2c4c-4c61-880b-d0f5711fd3b6_calibrate_947142_2.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9bee2f9a-97d3-4347-a2f8-7815758a2f53_characterizeImage_947142_2.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 143af9b46a2249118eb254c8f19c7962 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 3fdf6cdf2484ae3c602ec9833c84de9e parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c4e3c8fc-4d9e-4467-b5e1-a876b6af0bb2_calibrate_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c4e3c8fc-4d9e-4467-b5e1-a876b6af0bb2_calibrate_947140_28.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8503d3c4-2094-4389-8761-fd3eabe3757a_characterizeImage_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8503d3c4-2094-4389-8761-fd3eabe3757a_characterizeImage_947140_28.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f40914c65f09e05209777f30898599d with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 3d6196887ab180746bed53c042af5838 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/42a28457-44ae-4c48-a33d-74c139343c55_calibrate_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/42a28457-44ae-4c48-a33d-74c139343c55_calibrate_947142_39.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/da69d1b2-9c82-4ec5-9bbc-e60e71df2707_characterizeImage_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/da69d1b2-9c82-4ec5-9bbc-e60e71df2707_characterizeImage_947142_39.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 580442d1fb0aa95fd2e78a735555c99d with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 5b0f115efb9c9205356e460c68808d74 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/11a81df3-4307-4813-ace9-a81687309885_calibrate_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/11a81df3-4307-4813-ace9-a81687309885_calibrate_947142_51.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/ea73a511-331d-4d26-b44e-9b4b7ca44fe4_characterizeImage_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/ea73a511-331d-4d26-b44e-9b4b7ca44fe4_characterizeImage_947142_51.stderr parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a09860ed146ea1a56eb8e26f55a0be3 with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash cf6e87e03d4ed6e71a172c8c2c9fd63b parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4c232da1-7d1b-4fea-9ee4-56e5462bebf3_calibrate_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4c232da1-7d1b-4fea-9ee4-56e5462bebf3_calibrate_947246_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/882ca2d2-d8b7-4eab-8b73-a8c6244175e1_characterizeImage_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/882ca2d2-d8b7-4eab-8b73-a8c6244175e1_characterizeImage_947246_43.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0385a24d42fd4ea527d0d129a997d67c with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 514cd33e4512b2fe2e7fe1257d58f5bf parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a94400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd0e39d1-e90c-4c61-97ff-5f8a9d5a6826_calibrate_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd0e39d1-e90c-4c61-97ff-5f8a9d5a6826_calibrate_947244_1.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93e77fbd-abdb-4eff-bec5-b18ce4e61ae2_characterizeImage_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93e77fbd-abdb-4eff-bec5-b18ce4e61ae2_characterizeImage_947244_1.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a726b75e81596e12d1569a4b49de124 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 0444271f46432f864fa6329d877ec5cc parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/78c5c431-ad84-431f-8a4f-0e6260b2c4ef_calibrate_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/78c5c431-ad84-431f-8a4f-0e6260b2c4ef_calibrate_947248_56.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/80ba1e19-612f-49c8-adc7-c4aeab104cbf_characterizeImage_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/80ba1e19-612f-49c8-adc7-c4aeab104cbf_characterizeImage_947248_56.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 118f63f541a451c9928b45e090b8f0ee with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 270c499e72e8cdfac04f6f1a4dbeef8f parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1785eb67-1434-4a1b-b9c5-726f181690fc_calibrate_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1785eb67-1434-4a1b-b9c5-726f181690fc_calibrate_947144_4.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6fe6b9c6-18ee-416c-88a1-dac3e4305ddd_characterizeImage_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6fe6b9c6-18ee-416c-88a1-dac3e4305ddd_characterizeImage_947144_4.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee834de46fde9ab4716a48363f50028 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash d027a8d157ce52e0c591d56350548441 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3a9341d3-94f0-4fe7-b1f0-88f9482c95db_calibrate_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3a9341d3-94f0-4fe7-b1f0-88f9482c95db_calibrate_947244_5.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/56d108d5-fde0-4e0c-9cb4-a7f4ffcd07a0_characterizeImage_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/56d108d5-fde0-4e0c-9cb4-a7f4ffcd07a0_characterizeImage_947244_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 0 failed parsl.dataflow.dflow INFO: Task 106 marked for retry parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash e1aa76fc976e759bb6925a40adeaac7e parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 1 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1114 try 0 failed parsl.dataflow.dflow INFO: Task 1114 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 5e025ed1bffea5938983458db9634c68 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6636480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 1 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 277 try 0 failed parsl.dataflow.dflow INFO: Task 277 marked for retry parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash c80c0e49b3d780c98429de7a18ddcae3 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 1 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 538 try 0 failed parsl.dataflow.dflow INFO: Task 538 marked for retry parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 0c6e7e67c3bd80fd9d002b7037abcc8c parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 1 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1012 try 0 failed parsl.dataflow.dflow INFO: Task 1012 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 062fa3537fb08c68e0ba8aa84e49ca9d parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:03 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Parsl task 1012 try 1 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 922 try 0 failed parsl.dataflow.dflow INFO: Task 922 marked for retry parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 7c54a11ac66d5869c2858937caf78e0a parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 1 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 940 try 0 failed parsl.dataflow.dflow INFO: Task 940 marked for retry parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 7765451958e22ffbd9ed4848351d1279 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 1 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 83 try 0 failed parsl.dataflow.dflow INFO: Task 83 marked for retry parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 8d0d2b16d263d66e2a8fd7940a07fb6a parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 1 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 534e6629004402dc2d4ce9c58e5c7173 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 9265b8755fde1080a652659935443f6b parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c0fec11-9702-4554-bfd9-d09911cf8103_calibrate_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c0fec11-9702-4554-bfd9-d09911cf8103_calibrate_947140_14.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d5e8eb69-5ea2-4c60-b60e-300f8aafe6fa_characterizeImage_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/d5e8eb69-5ea2-4c60-b60e-300f8aafe6fa_characterizeImage_947140_14.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e316351d402ac61eda30b80a88c1f0b3 with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash e96d0bbee2a1c933b60b3e06df484c81 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f6e4a225-151f-4402-873a-43263d897b8a_calibrate_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f6e4a225-151f-4402-873a-43263d897b8a_calibrate_947144_7.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/db2665cc-f1ff-4c49-bef5-cdae5f473a34_characterizeImage_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/db2665cc-f1ff-4c49-bef5-cdae5f473a34_characterizeImage_947144_7.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd1840cf99dffaed63d627df6ed9d802 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash d78365b56b2c413451168c3a595ff3d1 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d343eec3-0b01-4b8c-b08c-7bd17798e673_calibrate_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d343eec3-0b01-4b8c-b08c-7bd17798e673_calibrate_947140_17.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/531db1c2-6f99-4cd3-998e-f9b9eec9d899_characterizeImage_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/531db1c2-6f99-4cd3-998e-f9b9eec9d899_characterizeImage_947140_17.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a45c14e655a253635d663241a8551fb with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash e2dd8871c7333ac08865771a94cc0cb4 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/6aa28346-f67a-4f63-acc9-3374d1713b7e_calibrate_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/6aa28346-f67a-4f63-acc9-3374d1713b7e_calibrate_947144_57.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e56fafc7-4755-457b-b3cb-16b521023f69_characterizeImage_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e56fafc7-4755-457b-b3cb-16b521023f69_characterizeImage_947144_57.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d5d376fd9d42c700a8f5f4c5eb612d with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash a86acb109b8163ecaa54bdbc59c3bed6 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/38355f3e-582c-49fb-8b9d-43d5d4f4c365_calibrate_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/38355f3e-582c-49fb-8b9d-43d5d4f4c365_calibrate_947144_34.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6b540640-a0a7-4a71-95aa-dbbc4cc0e4a2_characterizeImage_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/6b540640-a0a7-4a71-95aa-dbbc4cc0e4a2_characterizeImage_947144_34.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7aa788e7d3a41a489810fff83c51425f with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ba07c121-d5da-40b3-ae69-f68a76a69c79_calibrate_947140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ba07c121-d5da-40b3-ae69-f68a76a69c79_calibrate_947140_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 304 try 1 failed parsl.dataflow.dflow ERROR: Task 304 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 247e7560f1f011c3aba1b0f695838e83 with result from task 304 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 305 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 305 try 0 failed parsl.dataflow.dflow INFO: Task 305 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f6223594-e7bf-4bf4-9d8f-0b63358418a2_calibrate_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f6223594-e7bf-4bf4-9d8f-0b63358418a2_calibrate_947142_61.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5c5f838a-e672-4155-99c2-ee35c3d3e98d_characterizeImage_947142_61.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe42cd72aed92e4df6d7e1d52c1bc93 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 9b98df980563fcfbc93dcbd4d8293ee9 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9ce3aa21-4ecb-4a2c-82b6-b7845add6d41_calibrate_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9ce3aa21-4ecb-4a2c-82b6-b7845add6d41_calibrate_947144_13.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/2e085133-842c-4701-866c-0ce12dca1e30_characterizeImage_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/2e085133-842c-4701-866c-0ce12dca1e30_characterizeImage_947144_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 559 try 0 failed parsl.dataflow.dflow INFO: Task 559 marked for retry parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 5335be9eb348878bc7cf831886262e84 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 1 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 535 try 0 failed parsl.dataflow.dflow INFO: Task 535 marked for retry parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 81d0c67e33972fe2bec470382f84ff9f parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 1 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1109 try 0 failed parsl.dataflow.dflow INFO: Task 1109 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 86821a3b104098cb7a5666332e1a46f2 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6635c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 1 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 224 try 0 failed parsl.dataflow.dflow INFO: Task 224 marked for retry parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 2cd839efb5ccf1cd44bb356b45b5c956 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 1 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 167 try 0 failed parsl.dataflow.dflow INFO: Task 167 marked for retry parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 652938195e0269f9fe25c9a101453fc9 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 1 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 557 try 0 failed parsl.dataflow.dflow INFO: Task 557 marked for retry parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash ac09530b4e37b672a12fc994f5237ecb parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 1 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 272 try 0 failed parsl.dataflow.dflow INFO: Task 272 marked for retry parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash fbbf13947059badee0825443842af258 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 1 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 530 try 0 failed parsl.dataflow.dflow INFO: Task 530 marked for retry parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 3c8e4edfe83177e3725ba6367fd9693e parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 1 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb9439c2a5a6d46c534fb33f54ead33 with result from task 1039 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:04 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash dab047863ae903ded10785ed761c629a parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7ade571c-536b-431c-b9e3-ca5e6ee96cfb_calibrate_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7ade571c-536b-431c-b9e3-ca5e6ee96cfb_calibrate_947142_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a77b302e-bd62-4ec3-bb57-b9bcb691e54b_characterizeImage_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a77b302e-bd62-4ec3-bb57-b9bcb691e54b_characterizeImage_947142_21.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91e2761600c16eb5b241f37acd328db1 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash e0d86766f3c23d38761d38edb2156022 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/af23dc0c-4772-4417-ba8f-368bc961a4d4_calibrate_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/af23dc0c-4772-4417-ba8f-368bc961a4d4_calibrate_947140_59.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cfb8e066-89c9-4ca6-85ed-c79148987f8c_characterizeImage_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/cfb8e066-89c9-4ca6-85ed-c79148987f8c_characterizeImage_947140_59.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee87bb2ff6440f47d4942984c58e27db with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash b7ea23751c46e057c09d44e2243912e9 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/808b1202-d59c-4dd0-950e-2eb74018723f_calibrate_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/808b1202-d59c-4dd0-950e-2eb74018723f_calibrate_947244_22.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0b5f3abb-1e31-4f85-91ee-46182a7a79b8_characterizeImage_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0b5f3abb-1e31-4f85-91ee-46182a7a79b8_characterizeImage_947244_22.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f12927a2898aadcde17ca131e8ee2a with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 2a375b60dfd0082a42308eb4454ca35b parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0bffa731-dda1-4f01-a802-444f86957dc0_characterizeImage_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0bffa731-dda1-4f01-a802-444f86957dc0_characterizeImage_947144_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/1e5c22fd-eb3a-468d-9598-daf6634fbb37_isr_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/isr/947144/1e5c22fd-eb3a-468d-9598-daf6634fbb37_isr_947144_22.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42b57d072d4303c588e7a64d722f08af with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 1eae75f6b41763dbc4716d0fe4e36285 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c983af75-9aa5-4eb6-a455-f7ef3439af48_calibrate_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c983af75-9aa5-4eb6-a455-f7ef3439af48_calibrate_947140_21.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/df45d007-1503-42a2-afa3-0022c8a93b30_characterizeImage_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/df45d007-1503-42a2-afa3-0022c8a93b30_characterizeImage_947140_21.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f331e11f9b9c626b64a4c71e2751182f with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/11d2dd37-aacb-46c1-a418-df32638043dd_calibrate_947140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/11d2dd37-aacb-46c1-a418-df32638043dd_calibrate_947140_23.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e6670c18a3ec4ffe1dc684b1f76eb7 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fb1176c5-76ce-4726-9fde-a0d60657e385_calibrate_947140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fb1176c5-76ce-4726-9fde-a0d60657e385_calibrate_947140_33.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d323208259ea707f930f69a665abebe with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1af99d65-9342-4b33-b228-50d96dc74316_calibrate_947144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1af99d65-9342-4b33-b228-50d96dc74316_calibrate_947144_31.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a66821942145c731305fe207bf60735 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/30fa45d1-5082-469f-9666-b3978ffc465e_calibrate_947140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/30fa45d1-5082-469f-9666-b3978ffc465e_calibrate_947140_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:04 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdb13fd5bcf6b494f2fa81b0fe8a6534 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash e09143dd1fbac01594e0a263c72f0f43 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b15aee3d-93de-49c7-97b1-258890adfd63_calibrate_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b15aee3d-93de-49c7-97b1-258890adfd63_calibrate_947140_46.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c9d6b588-c571-4082-927a-a6160d612744_characterizeImage_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c9d6b588-c571-4082-927a-a6160d612744_characterizeImage_947140_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:05 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:05 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0643f98d1a11cb5604870dff8def46 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3139b048-d5f3-4903-af08-7f1232229b9f_calibrate_947140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3139b048-d5f3-4903-af08-7f1232229b9f_calibrate_947140_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:05 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:05 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff473303940088f21621b682d4e2b184 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b987c074-cc20-4799-9a03-64680f967eed_calibrate_947142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b987c074-cc20-4799-9a03-64680f967eed_calibrate_947142_8.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1964280b64c98a40dd1b7eba64d3ab with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/70fd144a-798f-4d2c-b86f-8d75a2944853_calibrate_947140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/70fd144a-798f-4d2c-b86f-8d75a2944853_calibrate_947140_6.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14c679ff8e9af33e3b24391e8d64bb55 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash ef0fb4c319f862523c31056f5a67e37c parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fe085fad-5c99-4306-8fdf-196a13cde152_calibrate_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fe085fad-5c99-4306-8fdf-196a13cde152_calibrate_947246_51.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0d0ccf5c-2d78-4e5b-a83d-50a2f12b09be_characterizeImage_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0d0ccf5c-2d78-4e5b-a83d-50a2f12b09be_characterizeImage_947246_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:06 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac0a49e766d737e7c61cace397d9e562 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/81b49969-4d65-45c7-a3d9-6dc4c843db89_calibrate_947140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/81b49969-4d65-45c7-a3d9-6dc4c843db89_calibrate_947140_49.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1429d7a919069cbe4f517b57fa63b56 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/68b375b7-9386-4f0a-8c2d-a380d962800d_calibrate_947142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/68b375b7-9386-4f0a-8c2d-a380d962800d_calibrate_947142_58.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 739878c69c522942b83067366216c146 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c1fc705f-c0e7-4a76-881f-ff61592cda51_calibrate_947140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c1fc705f-c0e7-4a76-881f-ff61592cda51_calibrate_947140_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:06 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:07 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0422cefcf2b34bca3da598ff2c45bcca with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash f9804de8c14110d669ecc8139fcfd3c8 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/95aaf963-039b-4e90-b973-dddfa6049190_calibrate_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/95aaf963-039b-4e90-b973-dddfa6049190_calibrate_947140_3.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0181f7bc-e344-48c0-aa0e-5cdc99835f56_characterizeImage_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0181f7bc-e344-48c0-aa0e-5cdc99835f56_characterizeImage_947140_3.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3713eb33abde841a0ba05e5e29fb89ec with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b855ccd-7fac-4667-92cf-823c430b8939_calibrate_947140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b855ccd-7fac-4667-92cf-823c430b8939_calibrate_947140_8.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2366f4f091b69d8c6b8f35ddaff78c40 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 21100d541d71393c51fb633acf371db1 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/a85edb93-443c-4065-92df-c5a4a9c23541_calibrate_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/a85edb93-443c-4065-92df-c5a4a9c23541_calibrate_947140_42.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/9649a5cf-79a1-4987-a45a-33c8e383fe0f_characterizeImage_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/9649a5cf-79a1-4987-a45a-33c8e383fe0f_characterizeImage_947140_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:07 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0106c5ba4f7390651b0f7067724aef3 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash c2dbd060e2a0ec1616da1272155ec276 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/60dba422-8dcc-4959-9e0f-73507e557e52_calibrate_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/60dba422-8dcc-4959-9e0f-73507e557e52_calibrate_947244_33.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/00da1f83-3245-490d-81d5-c6dbbd0cf03a_characterizeImage_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/00da1f83-3245-490d-81d5-c6dbbd0cf03a_characterizeImage_947244_33.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1c910a05d5678cb334310dd374ac6c with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c9353a11-d125-4d83-a07b-181d988653a9_calibrate_947140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c9353a11-d125-4d83-a07b-181d988653a9_calibrate_947140_31.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e075e39bb9507a015e21d1197b93d9 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b59c451-8d66-434a-8301-b36d91865e7d_calibrate_947140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/2b59c451-8d66-434a-8301-b36d91865e7d_calibrate_947140_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:07 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76b72ce5dc9478df07097f65b87c32c0 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 305afc30d587db2bab1b673ce8c66e14 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/19fca630-e4a4-4af5-8c79-49fa018c2e28_calibrate_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/19fca630-e4a4-4af5-8c79-49fa018c2e28_calibrate_947248_38.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7b8b129d-1e38-4baa-be13-4fb4c468349c_characterizeImage_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7b8b129d-1e38-4baa-be13-4fb4c468349c_characterizeImage_947248_38.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e879e0c6930c89345d07f1a9c60a25b with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0de8c5da-770d-4212-8f83-862da84d97cb_calibrate_947142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0de8c5da-770d-4212-8f83-862da84d97cb_calibrate_947142_28.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1ea49b71ee50469c6cae3c4485e91cf with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 4e4c9b1fad96a997c983f5100706c6d4 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/6643f9e5-9ab0-4349-9063-45aff6785571_calibrate_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/6643f9e5-9ab0-4349-9063-45aff6785571_calibrate_947140_48.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8d1ce7c1-bdd7-4fb1-ae74-f7a7dfee810a_characterizeImage_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8d1ce7c1-bdd7-4fb1-ae74-f7a7dfee810a_characterizeImage_947140_48.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5e381853cacf675b80c9c13fdfa40a with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/455d8ce8-4d74-49dc-ad21-766b3b17e87e_calibrate_947142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/455d8ce8-4d74-49dc-ad21-766b3b17e87e_calibrate_947142_5.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da24c4fb20d40d57f41100edb6a4af4f with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2b045656-a550-4968-8188-2c8a28be52de_calibrate_947142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2b045656-a550-4968-8188-2c8a28be52de_calibrate_947142_19.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e10b227402599a2d0df803789a61d58 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b4ac2cb-8e3d-41ab-ae59-b41f848335c3_calibrate_947140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b4ac2cb-8e3d-41ab-ae59-b41f848335c3_calibrate_947140_57.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e46ce81b0634d6219ca1736b3e9fa7 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc0981a1-33fe-4378-ab51-ae2a2035c168_calibrate_947140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc0981a1-33fe-4378-ab51-ae2a2035c168_calibrate_947140_20.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b85f4f22d707d9f17a9a6f92a09b250 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c49f566a-f305-4821-a4f5-4e5e14c845c6_calibrate_947140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c49f566a-f305-4821-a4f5-4e5e14c845c6_calibrate_947140_27.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c541ecfb903e52bcb6d5b9a97e82e4b1 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8d3074bf-7cfe-41d1-bafe-e385cae2b38b_calibrate_947140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8d3074bf-7cfe-41d1-bafe-e385cae2b38b_calibrate_947140_43.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9d3afb6b73f99c2f4fa292994290eb with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/28713079-9e9e-4445-bf18-c1d94b86eccf_calibrate_947140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/28713079-9e9e-4445-bf18-c1d94b86eccf_calibrate_947140_30.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08fd4fc8961b8a8fbc8df45517979cb1 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 291b922fcad1625995cd21031ddd53cb parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c7eae140-d3c0-4473-ab09-ec5ca328f99f_calibrate_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c7eae140-d3c0-4473-ab09-ec5ca328f99f_calibrate_947142_13.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3ee9c111-f99e-4937-b8ac-4f15d1a3d2e8_characterizeImage_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3ee9c111-f99e-4937-b8ac-4f15d1a3d2e8_characterizeImage_947142_13.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6417472c9dfbce20848e974c343e4904 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash ea4f60f212bb093aa9dd552e66efbcc5 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/4c4112a9-9e30-4a99-8057-1f3b5a6a1a39_calibrate_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/4c4112a9-9e30-4a99-8057-1f3b5a6a1a39_calibrate_947142_60.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2339d140-3397-413a-a288-52ca9a8c1e59_characterizeImage_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2339d140-3397-413a-a288-52ca9a8c1e59_characterizeImage_947142_60.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78af5e8ca6b7f3792e1ea54150e6a8f7 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/93f481ea-d893-435e-a903-a095d61bf900_calibrate_947144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/93f481ea-d893-435e-a903-a095d61bf900_calibrate_947144_11.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97566666664a7aeba021bd909782e4a7 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d74538eb-bec4-495b-9999-0e6d4f0a7cf3_calibrate_947140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d74538eb-bec4-495b-9999-0e6d4f0a7cf3_calibrate_947140_50.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 129c391453da4c16f9e134f03a3830a7 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/362eaf66-b47c-47cd-a915-fdeaf52f39a7_calibrate_947140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/362eaf66-b47c-47cd-a915-fdeaf52f39a7_calibrate_947140_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 928 try 1 failed parsl.dataflow.dflow ERROR: Task 928 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f90cfdff87e00ff246e68340692c541e with result from task 928 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 929 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 929 try 0 failed parsl.dataflow.dflow INFO: Task 929 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dc6e126e-d325-4eee-ab5e-81d8a5e88c76_calibrate_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dc6e126e-d325-4eee-ab5e-81d8a5e88c76_calibrate_947144_2.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/7deb8690-49af-450d-8b3d-138df80a99c0_characterizeImage_947144_2.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df1c6ae4fcbdabb9532616da34d26e98 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 9905ad05596b7aa0f51e30f980284a7d parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/51a19127-29c4-40f1-9305-cd8c5bc1cbad_calibrate_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/51a19127-29c4-40f1-9305-cd8c5bc1cbad_calibrate_947140_11.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8bc3643a-4c89-4759-b9e0-e3b204cc0c9b_characterizeImage_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8bc3643a-4c89-4759-b9e0-e3b204cc0c9b_characterizeImage_947140_11.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6263748383ec4c1b2eb0ce20ff9b797 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9214c3ef-560a-474b-b319-c9cc413a6127_calibrate_947144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9214c3ef-560a-474b-b319-c9cc413a6127_calibrate_947144_54.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7c257450d74316fdcf02521e46d1dac with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/56e5eae4-efd7-48c1-8ffe-26592430857b_calibrate_947142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/56e5eae4-efd7-48c1-8ffe-26592430857b_calibrate_947142_15.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f19facd227b206a2d196ded146d8bdb with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ac68a730-0c0d-48b9-8894-afcadc356080_calibrate_947140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ac68a730-0c0d-48b9-8894-afcadc356080_calibrate_947140_52.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d780c87d9ef0b0446a6e04f5bc37ad9 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 0034d1b4fcd82cf0e2bbb51c567e5d23 parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c8d22479-4606-426e-9735-84adea0d76e0_calibrate_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c8d22479-4606-426e-9735-84adea0d76e0_calibrate_947142_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/dbd761ce-b293-4a82-8977-752bd018fab8_characterizeImage_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/dbd761ce-b293-4a82-8977-752bd018fab8_characterizeImage_947142_32.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee76e12d233d5011ccad13d4fa94d3a with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/75be186b-b81e-45a7-8ad8-96411d2d595a_calibrate_947140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/75be186b-b81e-45a7-8ad8-96411d2d595a_calibrate_947140_19.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec9a57a3416e829e1ac7b67f4b697171 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash af6a7dab2463fbf746aa9b1fc2574d9b parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:12 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8aa156ea-9810-4421-88a0-9fe0b2b85188_calibrate_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8aa156ea-9810-4421-88a0-9fe0b2b85188_calibrate_947142_43.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/783d843a-48fc-4268-9ba9-748730a9f1e2_characterizeImage_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/783d843a-48fc-4268-9ba9-748730a9f1e2_characterizeImage_947142_43.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb9d9ef6ca1c8dbeec8989829b9030b with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/05c7c642-fdc2-476c-af84-95082cb07f80_calibrate_947144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/05c7c642-fdc2-476c-af84-95082cb07f80_calibrate_947144_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:12 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e663974e9a0db967cea6b3403a752d with result from task 137 parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:12 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f87f1c54-6d96-479a-a0ad-3b0e7350eb83_calibrate_947142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f87f1c54-6d96-479a-a0ad-3b0e7350eb83_calibrate_947142_47.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcd77ae6c7502fdb215dafa197f2e053 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 03367eb0b4bf47062575ab8e749fcaf6 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/effb0ac7-c82b-4723-9209-4f457c285484_calibrate_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/effb0ac7-c82b-4723-9209-4f457c285484_calibrate_947142_20.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8735c15d-2c23-4048-b86c-e4780f4b73fa_characterizeImage_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8735c15d-2c23-4048-b86c-e4780f4b73fa_characterizeImage_947142_20.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d5670453a0e23dc9bba0db1a4a63d7 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 70507ae2b31232ef2532ae173ca85b5a parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/cfd97af0-c520-419c-8d33-2fb87286854d_calibrate_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/cfd97af0-c520-419c-8d33-2fb87286854d_calibrate_947246_41.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d10b97d9-3010-433a-9df5-b708991e751c_characterizeImage_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d10b97d9-3010-433a-9df5-b708991e751c_characterizeImage_947246_41.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b9d25c75d9907d5da371bbf2107113 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/585c66a6-4927-41df-8583-499ddc6cd4b5_calibrate_947142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/585c66a6-4927-41df-8583-499ddc6cd4b5_calibrate_947142_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:13 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b67c07f3551a336b1e335d38340ddf7 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 73891e9846e4de07dd31e0094cff420c parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/aafa1946-1fd0-4bdb-9d47-244752c6873e_calibrate_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/aafa1946-1fd0-4bdb-9d47-244752c6873e_calibrate_947140_5.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f3ed0f45-cc4a-4266-b605-c73c95fb43c0_characterizeImage_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f3ed0f45-cc4a-4266-b605-c73c95fb43c0_characterizeImage_947140_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:13 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aae658d3fcbfcdf6105d772f4a007af1 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cebd914a-6bf1-4c05-a1b6-94f1f3a8e364_calibrate_947142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cebd914a-6bf1-4c05-a1b6-94f1f3a8e364_calibrate_947142_31.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0369a41fee84cfcf726cd8054d4e8d6c with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 2bdb9f72d2e6930e39bb4d257931893e parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/29da6807-6727-4a32-9b19-a926bf18dcca_calibrate_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/29da6807-6727-4a32-9b19-a926bf18dcca_calibrate_947144_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/280c5f24-4da2-41ff-a401-ba1ceded3630_characterizeImage_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/280c5f24-4da2-41ff-a401-ba1ceded3630_characterizeImage_947144_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:14 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8bb004d27557a52bd376b1704f935ac with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/323177fe-06ae-4fae-a35f-2750c186f34b_calibrate_947142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/323177fe-06ae-4fae-a35f-2750c186f34b_calibrate_947142_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:14 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe3ca3971d36385f1b1233174677e8c2 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/7b37012e-ef90-48d4-bf52-fefeab65d4e4_calibrate_947144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/7b37012e-ef90-48d4-bf52-fefeab65d4e4_calibrate_947144_23.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d65c61bf3b70affb1d2a0618243cb34d with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 5f4c77c425c53515642daff093f06d59 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5cc0746b-d469-4a80-a242-bc5cefc97403_calibrate_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5cc0746b-d469-4a80-a242-bc5cefc97403_calibrate_947142_27.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8a3fbcd9-7dc4-46d1-a150-3f699766aa65_characterizeImage_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/8a3fbcd9-7dc4-46d1-a150-3f699766aa65_characterizeImage_947142_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:14 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92011974b719a660bdfde311010566ca with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/950dadac-70cf-455d-8c6a-4070b8dbffeb_calibrate_947142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/950dadac-70cf-455d-8c6a-4070b8dbffeb_calibrate_947142_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:15 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb5a7c6fcfced45d013d20a875708a33 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3a7ccf28-0f9b-418e-a1b8-c577bda7de6e_calibrate_947142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3a7ccf28-0f9b-418e-a1b8-c577bda7de6e_calibrate_947142_33.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 898331e8f1cc42bd323dc6ffa57a6316 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash b07679e01b7f3f8b103a004d28d96b04 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2efc6364-f06c-4fef-90c7-d6381cd33b12_calibrate_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2efc6364-f06c-4fef-90c7-d6381cd33b12_calibrate_947144_10.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f778265d-805d-4589-9d25-a114c256b209_characterizeImage_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f778265d-805d-4589-9d25-a114c256b209_characterizeImage_947144_10.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e093a9fb313e2ef82b6d7d645bd8f83 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 08810dae0b1f40792c83ab9033b1cacc parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/096055ee-8724-41b2-8b71-0de685f39dc7_calibrate_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/096055ee-8724-41b2-8b71-0de685f39dc7_calibrate_947140_38.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19cca926-4ca5-4359-b550-35b03501e94c_characterizeImage_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19cca926-4ca5-4359-b550-35b03501e94c_characterizeImage_947140_38.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03d74547b3bffa077458d60f708a42a9 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 788b423ac851b9fc6a5a080bd9a8f923 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/575a5b63-2ee9-45a0-89ed-dfcfa9c9a9b8_calibrate_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/575a5b63-2ee9-45a0-89ed-dfcfa9c9a9b8_calibrate_947142_42.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/950ea26a-1ed4-447b-8a90-b6019137f296_characterizeImage_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/950ea26a-1ed4-447b-8a90-b6019137f296_characterizeImage_947142_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:15 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:16 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:16 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1949662c5cc518ae9bec1aa02dfc72ec with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8bd4b43d-09f8-4b08-8b20-b01933ce7e07_calibrate_947144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8bd4b43d-09f8-4b08-8b20-b01933ce7e07_calibrate_947144_39.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8796730398f8047b67e672d623409f70 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/230582ab-3291-4b72-a902-8edeb65950c8_calibrate_947144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/230582ab-3291-4b72-a902-8edeb65950c8_calibrate_947144_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:17 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01cb633c003fa3efbf97c3effd72b2dc with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/447dcf36-786f-497f-a0fe-740b7ba06c64_calibrate_947142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/447dcf36-786f-497f-a0fe-740b7ba06c64_calibrate_947142_55.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3deea67ca488ad6d77b809d026254442 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/397fe74d-d653-4aa8-99aa-07f5d2db2122_calibrate_947144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/397fe74d-d653-4aa8-99aa-07f5d2db2122_calibrate_947144_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:17 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 772 try 1 failed parsl.dataflow.dflow ERROR: Task 772 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 17108cb988fa9bfbc8816f60b7972700 with result from task 772 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 773 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 773 try 0 failed parsl.dataflow.dflow INFO: Task 773 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b591d46-03ca-4d58-af7f-72162342ce09_calibrate_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1b591d46-03ca-4d58-af7f-72162342ce09_calibrate_947140_61.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/32dba308-afa6-4266-8448-1f73d5ad7a68_characterizeImage_947140_61.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4de797befa17140595dce9f32797536c with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 974f873bdeaa927146ad7dbe6cafd42d parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c6a8951e-bf4e-46f4-a0ed-d0cfea21098d_calibrate_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c6a8951e-bf4e-46f4-a0ed-d0cfea21098d_calibrate_947140_26.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/638c986a-ef14-4933-9060-fdd130d71183_characterizeImage_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/638c986a-ef14-4933-9060-fdd130d71183_characterizeImage_947140_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:17 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d7875e35c3197f6135519b13667b52c with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e56b921c-50be-4b98-8fd6-a2abb94b0fc7_calibrate_947142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e56b921c-50be-4b98-8fd6-a2abb94b0fc7_calibrate_947142_59.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8becbbc6f437c96624f9417ebcd3e03b with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a286be51-e3a9-4822-98ad-9baf2a6b7075_calibrate_947144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a286be51-e3a9-4822-98ad-9baf2a6b7075_calibrate_947144_46.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1770bf4b44a6ee2e7854d8fc4a6ae675 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b2fb7630-2f91-4b4f-8ff1-e806cc2be4b7_calibrate_947142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/b2fb7630-2f91-4b4f-8ff1-e806cc2be4b7_calibrate_947142_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:18 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06c23c123f6f8f2992c7896f5eb0d728 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/b4e8a9b9-00e7-4ea4-bd7a-f16067ac80eb_calibrate_947144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/b4e8a9b9-00e7-4ea4-bd7a-f16067ac80eb_calibrate_947144_50.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67c1ff18f8c83a275cab436f18e84e3b with result from task 65 parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:18 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/34d6df00-d83c-48fb-9ea8-917365a3c1aa_calibrate_947144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/34d6df00-d83c-48fb-9ea8-917365a3c1aa_calibrate_947144_18.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d3ce363ae17e4de6e1e6abf7b297c9 with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/860f7316-1205-4a42-8802-1cbf199382ae_calibrate_947142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/860f7316-1205-4a42-8802-1cbf199382ae_calibrate_947142_30.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc310fd40b923d0ad542dd95da92e2ae with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/fe2be2f5-7bc0-4dd7-8c14-e9745f21ce46_calibrate_947142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/fe2be2f5-7bc0-4dd7-8c14-e9745f21ce46_calibrate_947142_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:19 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb4520f86f78701a8e146f6ddfe27cec with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash f0b20c9056f530959fa72841f2b46004 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1a531107-4abe-4f45-a4dd-d9b9d0a4f90c_calibrate_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1a531107-4abe-4f45-a4dd-d9b9d0a4f90c_calibrate_947140_55.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/64cef954-7e1c-4430-9cce-719f883377db_characterizeImage_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/64cef954-7e1c-4430-9cce-719f883377db_characterizeImage_947140_55.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9337957960749982ccb7b6f0c6d1ec0f with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash dcea83eac7fa18950070a3902418680d parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68785e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dd949347-2f7e-4e06-8713-cb7819c9a359_calibrate_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dd949347-2f7e-4e06-8713-cb7819c9a359_calibrate_947144_3.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3e4c8f28-9dfb-4197-a3e9-890c2919676d_characterizeImage_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3e4c8f28-9dfb-4197-a3e9-890c2919676d_characterizeImage_947144_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:19 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08b3b875ed5a16e91ede7841fe69ae55 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash fa310b794145c46c9b5f643a81e434e4 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a96660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f337863-2ec1-4a82-bba6-9decdb4ec23a_calibrate_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f337863-2ec1-4a82-bba6-9decdb4ec23a_calibrate_947248_7.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/286d45ba-8e3a-471c-9d83-f5d89c4cf5d7_characterizeImage_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/286d45ba-8e3a-471c-9d83-f5d89c4cf5d7_characterizeImage_947248_7.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeaa8dc96851fb9b3c149cd6e8529649 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash a15d5377db0d06ccb9ee4ef4cf402c00 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a9c7c657-733d-48cb-ad33-0c49e50eb963_calibrate_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a9c7c657-733d-48cb-ad33-0c49e50eb963_calibrate_947144_59.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ff1e0be2-89ec-45e1-b4e1-0c6a3e2db220_characterizeImage_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ff1e0be2-89ec-45e1-b4e1-0c6a3e2db220_characterizeImage_947144_59.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d11463ab5d19ca7acfdcb39f61dc4c6 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash f88d76ee41cc5fad2d521e9b9713ef0c parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1e63a0f6-7c0a-4977-ba53-62b5ea0350ac_calibrate_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1e63a0f6-7c0a-4977-ba53-62b5ea0350ac_calibrate_947140_18.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/b4a62b0e-9c30-4efc-b0e5-7d630d0731a0_characterizeImage_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/b4a62b0e-9c30-4efc-b0e5-7d630d0731a0_characterizeImage_947140_18.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b0f30c6ff78fc006e91838b4a07adf2 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3625242b-0459-447a-95c2-740904dd2dfb_calibrate_947144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3625242b-0459-447a-95c2-740904dd2dfb_calibrate_947144_38.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d81fc38b2476c1442122ad131d62df0b with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ffb4d8d1-fe87-4520-aef5-d4d16add3f18_calibrate_947144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ffb4d8d1-fe87-4520-aef5-d4d16add3f18_calibrate_947144_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:20 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe0807f20c154dc4272039e52d091bc with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/441ce415-7fb6-4e81-80b5-523990766793_calibrate_947142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/441ce415-7fb6-4e81-80b5-523990766793_calibrate_947142_37.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd75de12110bea256490fc5297c2b2a6 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 2a72d9ab0d7645801633a3db377d346c parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/17bec250-732a-4283-a22b-97b3ec49688b_calibrate_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/17bec250-732a-4283-a22b-97b3ec49688b_calibrate_947142_36.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/36aa0089-a77b-4e69-b975-8013157313fa_characterizeImage_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/36aa0089-a77b-4e69-b975-8013157313fa_characterizeImage_947142_36.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8041bb5836e3a50ff38623ec09ce5fd0 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash a0bcd37c85ad1db307e7b7f887657aba parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68784a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5c4751a7-0930-4aa8-85da-f0baee89f5d1_calibrate_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5c4751a7-0930-4aa8-85da-f0baee89f5d1_calibrate_947142_22.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/4a3bd186-aa14-4779-8d29-8786a35f46cb_characterizeImage_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/4a3bd186-aa14-4779-8d29-8786a35f46cb_characterizeImage_947142_22.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bceab9fb4e429edf6618caade8d3649 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 2dde750cb26fa2a0c5f135809d45c40f parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0fe5128e-3caa-4a29-996e-18db426c4e49_calibrate_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0fe5128e-3caa-4a29-996e-18db426c4e49_calibrate_947142_57.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3a2bb230-8936-4b97-810e-a8ea4621543f_characterizeImage_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/3a2bb230-8936-4b97-810e-a8ea4621543f_characterizeImage_947142_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:20 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 336fb8a33b5d4bb6e98860627b87b4cc with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 0db88015e9e7c661a24aac9a90ca31c5 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41e29fd2-a722-4f58-9998-ac1e4af3262a_calibrate_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41e29fd2-a722-4f58-9998-ac1e4af3262a_calibrate_947144_20.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ed641de2-1f68-44e8-8268-738bef07aa4a_characterizeImage_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ed641de2-1f68-44e8-8268-738bef07aa4a_characterizeImage_947144_20.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25789eb1f24029ec96fd19f6111ac421 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 6eca72687cf739bb4e228727d4260597 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9ee2a0d7-04bd-4448-8b0e-16865eabdd8e_calibrate_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9ee2a0d7-04bd-4448-8b0e-16865eabdd8e_calibrate_947248_8.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/27023f8d-fa91-4998-9037-c41cee00af06_characterizeImage_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/27023f8d-fa91-4998-9037-c41cee00af06_characterizeImage_947248_8.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c002eb31c67344ec0d256a689500619 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e3b5f889-5f26-4feb-8ca4-1948984c4126_calibrate_947144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e3b5f889-5f26-4feb-8ca4-1948984c4126_calibrate_947144_51.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa0ca4dd83e54c4be1da92e8543473b9 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/119aaa9a-a03b-4e21-9141-77cd8693812d_calibrate_947144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/119aaa9a-a03b-4e21-9141-77cd8693812d_calibrate_947144_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:20 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1ead47c448b7707f94c0690ab566573 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/15b0de3b-053d-4e6e-9668-3676ab9200e7_calibrate_947142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/15b0de3b-053d-4e6e-9668-3676ab9200e7_calibrate_947142_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:21 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc9c35abf187bd13e1e79209c965a8dc with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash b74fbc4770e003a1e9f9e8fcda53358f parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/65a8c7f2-2ac9-4f2f-b388-f40e094b4261_calibrate_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/65a8c7f2-2ac9-4f2f-b388-f40e094b4261_calibrate_947248_27.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/943d926b-22da-41b1-8bd3-7d4ec728bd88_characterizeImage_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/943d926b-22da-41b1-8bd3-7d4ec728bd88_characterizeImage_947248_27.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02badb582ff9f4b2d8b082bf9290c2af with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/de282414-f679-4b22-9615-9c8014766248_calibrate_947144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/de282414-f679-4b22-9615-9c8014766248_calibrate_947144_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:21 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4c77fa581753ba50bc3d0733dad0501 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e96e251b-425c-4e33-988a-aa8ba011aaa0_calibrate_947144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e96e251b-425c-4e33-988a-aa8ba011aaa0_calibrate_947144_49.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2c546c25d46601025a844c38c394ff8 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash babb9ffe214a584941c84513185883a4 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8e9eed70-98c3-4719-8ebc-c575b76e6a27_calibrate_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8e9eed70-98c3-4719-8ebc-c575b76e6a27_calibrate_947140_24.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/80a9daf4-b3bb-40c7-9a29-4613b826d419_characterizeImage_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/80a9daf4-b3bb-40c7-9a29-4613b826d419_characterizeImage_947140_24.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f56ea74bc565b288b9fc2b7307508072 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/355aac0c-3bf8-4892-beec-fdecc7a5d9c9_calibrate_947144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/355aac0c-3bf8-4892-beec-fdecc7a5d9c9_calibrate_947144_5.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b43755f66c32b48abafa1ab9715cf0fd with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0273b344-f050-43ac-b4e3-19d111b5ad7e_calibrate_947142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0273b344-f050-43ac-b4e3-19d111b5ad7e_calibrate_947142_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:22 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed795ccebc4601e73a6e1c03d6d6222b with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1cbcf670-41ce-441e-ba4a-4d2b64053e7a_calibrate_947144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1cbcf670-41ce-441e-ba4a-4d2b64053e7a_calibrate_947144_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:22 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94beb4c1f28eaaf7df9c1a3b9c126d96 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/593b49dc-bf9e-4b0b-9544-571fe0123eef_calibrate_947144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/593b49dc-bf9e-4b0b-9544-571fe0123eef_calibrate_947144_41.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3c217fab5140ed70d95b45bc715dfc with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 7ae2bc283ba8931ef601e1f56679a925 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2aba5df2-cc01-4b04-bfad-b164d42200ab_calibrate_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2aba5df2-cc01-4b04-bfad-b164d42200ab_calibrate_947144_52.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/592560c0-365b-45b0-945b-7d207f161a8b_characterizeImage_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/592560c0-365b-45b0-945b-7d207f161a8b_characterizeImage_947144_52.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca3513b7ba4a6824630600fe8120d0e with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash ce4c834fee439d6680ca5928587d3d48 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c1a2f34-a22c-406d-be9a-f21d80e9b358_calibrate_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c1a2f34-a22c-406d-be9a-f21d80e9b358_calibrate_947140_36.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e28f692b-5c2a-462f-8290-ef5b6aede70c_characterizeImage_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/e28f692b-5c2a-462f-8290-ef5b6aede70c_characterizeImage_947140_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:23 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78c6529edf21b49b8a9555969f55e632 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash b21d7d19d7e5a9fc8b9cfdf461b4b9a8 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e1ce4e2-3116-4656-8725-049df9095e58_calibrate_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e1ce4e2-3116-4656-8725-049df9095e58_calibrate_947142_48.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/996fe5d2-9484-4796-8a55-c014ef874409_characterizeImage_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/996fe5d2-9484-4796-8a55-c014ef874409_characterizeImage_947142_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:23 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:23 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51a34e756a33051eb4bfa6c2ae3a7dc5 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash a1d228b1361d34607340d4dca1e7fe0a parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c728c013-56de-4f15-b09d-edf704c4316a_calibrate_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c728c013-56de-4f15-b09d-edf704c4316a_calibrate_947140_22.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/35d3e4c7-f275-4585-8ca9-5637a25c02b0_characterizeImage_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/35d3e4c7-f275-4585-8ca9-5637a25c02b0_characterizeImage_947140_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 304 parsl.dataflow.strategy DEBUG: Executor multi has 304 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:37:24 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:37:24 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:24 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff1a9e901263374d4eb613e228d9e4fb with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash d7b8a6af679f6a1e6fb1ebf8cb6938eb parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c45ae31e-a06a-4790-a17a-9986a9f66cea_calibrate_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c45ae31e-a06a-4790-a17a-9986a9f66cea_calibrate_947244_29.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/55a3c841-70a9-4f6e-a8c5-0b082e3821b2_characterizeImage_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/55a3c841-70a9-4f6e-a8c5-0b082e3821b2_characterizeImage_947244_29.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bffcf29fcca3ab7323d6db0d0e627d90 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d385b13d-510e-4279-b1e3-c838b2567ef2_calibrate_947142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d385b13d-510e-4279-b1e3-c838b2567ef2_calibrate_947142_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:25 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:25 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd107ac91dd244ef5908a8f2c7c33b2 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash cc33cb20edaef4eddde6b6254bd44a21 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6977240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/138563ab-627a-422d-9003-80c417780486_calibrate_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/138563ab-627a-422d-9003-80c417780486_calibrate_947248_23.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/18f1382d-1e08-40e8-bab2-864d0b8eea74_characterizeImage_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/18f1382d-1e08-40e8-bab2-864d0b8eea74_characterizeImage_947248_23.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3a3f5ea3a95ff33572fa42c6d67016 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/046ccdf4-8273-44f8-8f6e-517a909fa0d0_calibrate_947248_26.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/046ccdf4-8273-44f8-8f6e-517a909fa0d0_calibrate_947248_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:26 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d0fd4fa324eb709da448923adda9cf1 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 6dfc2aa30a26397874f5a1caba6ec659 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bad552e7-9e70-4e8f-865e-72bd8ea50813_calibrate_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bad552e7-9e70-4e8f-865e-72bd8ea50813_calibrate_947140_12.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f137adcf-a41b-4da2-ace6-b0d99ee47012_characterizeImage_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f137adcf-a41b-4da2-ace6-b0d99ee47012_characterizeImage_947140_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:26 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f5a32bf16e801953fc30f9166f1a046 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 6dd0b9ff36cb672ece7a8f736bc7e048 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d88bc845-5338-480d-adf5-a3e39a19aa14_calibrate_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d88bc845-5338-480d-adf5-a3e39a19aa14_calibrate_947144_60.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a6db2017-3019-4db6-9217-87ca388b9823_characterizeImage_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/a6db2017-3019-4db6-9217-87ca388b9823_characterizeImage_947144_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:27 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:27 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a050f33229ffc1453e057ef560163afb with result from task 328 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:27 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash b2d80fc55e8a14d8dde70c6bfe2cc38a parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/493681bd-5070-49f5-9b46-5eccb7c5bd67_calibrate_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/493681bd-5070-49f5-9b46-5eccb7c5bd67_calibrate_947142_10.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/1f8b188f-6914-4daf-bde4-05f80882feb8_characterizeImage_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/1f8b188f-6914-4daf-bde4-05f80882feb8_characterizeImage_947142_10.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f0c07593d49df5cf4725414841555a8 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/97ca5fd3-80c4-442f-a180-aa10de889fd7_calibrate_947246_27.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/97ca5fd3-80c4-442f-a180-aa10de889fd7_calibrate_947246_27.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e65af1337e8b9f1ae48fac22f216d578 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash a2055a17277becb844d3fc4d7137bece parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4a882b26-26c5-4bdc-bb8f-1b39779bf15c_calibrate_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4a882b26-26c5-4bdc-bb8f-1b39779bf15c_calibrate_947144_12.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fe38f9f9-5674-4dc9-860a-8f19a96bde89_characterizeImage_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/fe38f9f9-5674-4dc9-860a-8f19a96bde89_characterizeImage_947144_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:28 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e200fd3f81f4378f66dede93ad9086eb with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash c0d016317b8c51366bf6ccb9bebc2e50 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e81cb99-7ba8-4446-b0f7-47e970394772_calibrate_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e81cb99-7ba8-4446-b0f7-47e970394772_calibrate_947142_17.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d689a836-6193-4ed8-8203-7ba42874e78b_characterizeImage_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/d689a836-6193-4ed8-8203-7ba42874e78b_characterizeImage_947142_17.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edcc985e2a3641bddeb9aaa9adbecdd3 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 63057d9bf45d67c5a79456e417f6568e parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d6ce8e82-335a-49f3-b653-af3489b25335_calibrate_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d6ce8e82-335a-49f3-b653-af3489b25335_calibrate_947144_14.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/41492023-cc39-4ecd-9848-8ae647dab23a_characterizeImage_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/41492023-cc39-4ecd-9848-8ae647dab23a_characterizeImage_947144_14.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36b72231db3c67d8c903617575e4c5f9 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 77a62315cddf158d989001ac505e97c5 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fda35ca5-2068-4d33-9a0f-18d832bd5d29_calibrate_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fda35ca5-2068-4d33-9a0f-18d832bd5d29_calibrate_947140_32.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/5645d668-fbe2-4d51-85a6-2d2937043b8d_characterizeImage_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/5645d668-fbe2-4d51-85a6-2d2937043b8d_characterizeImage_947140_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:28 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f742e6e189f8b5c1ca46eef62b1d231 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash a615771cf4cf900a22163d01f66d1cb0 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/db4b072a-9224-4286-90c1-be8899cfd106_calibrate_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/db4b072a-9224-4286-90c1-be8899cfd106_calibrate_947144_36.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b10e417f-6381-4f5e-8837-607b7d95a3a5_characterizeImage_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b10e417f-6381-4f5e-8837-607b7d95a3a5_characterizeImage_947144_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:29 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 115ccbfa2f62886a30a6ec40ca24721e with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash e89a38cbb5f26255893f2e96e4812350 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/c6dd51a4-69a4-4440-9225-140a7cf0b061_calibrate_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/c6dd51a4-69a4-4440-9225-140a7cf0b061_calibrate_947144_58.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f01173c1-67fb-400b-bd27-86e01761d5bf_characterizeImage_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/f01173c1-67fb-400b-bd27-86e01761d5bf_characterizeImage_947144_58.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e248dba6a923af60b5b38571ef579e55 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2cbb9be2-d4db-42ad-aac6-42f67d3f1c0b_calibrate_947142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/2cbb9be2-d4db-42ad-aac6-42f67d3f1c0b_calibrate_947142_35.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f98769daa6c64b1dd656ee3497744bd0 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c495bae1-02d3-4d00-a561-ea6927c8bdcb_calibrate_947244_51.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c495bae1-02d3-4d00-a561-ea6927c8bdcb_calibrate_947244_51.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880de859be19b8f9378c661e6d2fbfc5 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 1c2c3d6a61609e39dbb55b6f1d74aad3 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3d111534-eb0a-4eee-b91f-5de26cfec652_calibrate_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3d111534-eb0a-4eee-b91f-5de26cfec652_calibrate_947246_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a7a3c4f8-9f5c-4d91-b934-f439cd1f17d8_characterizeImage_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a7a3c4f8-9f5c-4d91-b934-f439cd1f17d8_characterizeImage_947246_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:29 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:29 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 404a3a41868c95d8e251c4445aea6367 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/331c3021-2564-4862-9e50-3f3e1053a757_calibrate_947246_45.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/331c3021-2564-4862-9e50-3f3e1053a757_calibrate_947246_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:30 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7078d1a9763e5900bbcefee41570612 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash e6f88270e01a9643a76555f5e02dea5f parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7697d1c7-df75-4d69-b3b0-705d37227c87_calibrate_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7697d1c7-df75-4d69-b3b0-705d37227c87_calibrate_947244_28.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/07f3bdb1-90b3-465e-9f6c-ed569b1360de_characterizeImage_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/07f3bdb1-90b3-465e-9f6c-ed569b1360de_characterizeImage_947244_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:30 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16a6bdde378d2418f3aaa67930f970da with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 72fced475f992a32372b25a331db3689 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f63d816-7a76-4c5b-84ac-3207b679a88f_calibrate_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f63d816-7a76-4c5b-84ac-3207b679a88f_calibrate_947248_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4e3fcf5d-7986-49e8-9a88-d837582bded7_characterizeImage_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4e3fcf5d-7986-49e8-9a88-d837582bded7_characterizeImage_947248_47.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9650a8f56c781fcd2c766f0ce52f23f with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash dcf6dfaf97e1103345f060cdec636877 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/54b4066d-f380-496f-b0e1-61564bb9002d_calibrate_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/54b4066d-f380-496f-b0e1-61564bb9002d_calibrate_947140_15.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f06de00d-ebef-491a-aca1-b62477ce3f17_characterizeImage_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/f06de00d-ebef-491a-aca1-b62477ce3f17_characterizeImage_947140_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:31 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10fd4fa84e2670c639dbbac1844691c0 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ae5b0a24-6b5b-4bed-90b3-1219d41a9d78_calibrate_947144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/ae5b0a24-6b5b-4bed-90b3-1219d41a9d78_calibrate_947144_6.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d94fcf77a491c473370044db32ac31 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash a53b48eef9f672eb253fe64071c44389 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/36829a68-6c4b-49f8-8f8a-542262bf0e25_calibrate_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/36829a68-6c4b-49f8-8f8a-542262bf0e25_calibrate_947140_39.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/4f483d5c-7b62-4e69-8649-5d0ea74d9546_characterizeImage_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/4f483d5c-7b62-4e69-8649-5d0ea74d9546_characterizeImage_947140_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:31 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:32 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:32 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55c13bb8552ab4aed5142ae4f11583c5 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 7a24056210ccd856eceba4ca81f572be parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c210eafb-16bc-4c53-80f9-ef7742153656_calibrate_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c210eafb-16bc-4c53-80f9-ef7742153656_calibrate_947244_4.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/41b43010-0df3-4b96-8ac9-83ec47f8a49d_characterizeImage_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/41b43010-0df3-4b96-8ac9-83ec47f8a49d_characterizeImage_947244_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:32 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84fccc3768d4d864ac030dd50cf5cd50 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0fc905d9-72b2-4c7c-ad0c-b14335d5f812_calibrate_947248_5.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0fc905d9-72b2-4c7c-ad0c-b14335d5f812_calibrate_947248_5.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0380520d8027bc8a1ae516df88888655 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash a10f4ed8853f3a5e0770413eb4b4169b parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/851eddc7-0153-4409-b91f-78fb52c23c14_calibrate_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/851eddc7-0153-4409-b91f-78fb52c23c14_calibrate_947244_59.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/aee5a96e-ce53-4fe7-9d63-896cdfd8b1eb_characterizeImage_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/aee5a96e-ce53-4fe7-9d63-896cdfd8b1eb_characterizeImage_947244_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:33 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 125aae77f9c007d9b79a247e4d88b0bf with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ae56a859-391b-4805-a17e-c2c99f141e09_calibrate_947140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ae56a859-391b-4805-a17e-c2c99f141e09_calibrate_947140_60.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a224d0962d8ab618b7b17b07f96d2096 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash a8532e1f5fb24d2db9d2ce53a30dd546 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6974b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/6e8cf3a8-95a9-4a02-8111-845617457cdd_calibrate_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/6e8cf3a8-95a9-4a02-8111-845617457cdd_calibrate_947244_49.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5ad1e235-91ba-4db6-ae4b-f577ed9a535d_characterizeImage_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5ad1e235-91ba-4db6-ae4b-f577ed9a535d_characterizeImage_947244_49.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b98a088c5d85c50a1d608a6cc37346b9 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2c8bcced-80c2-48c2-bc9a-2590e36ec6e3_calibrate_947246_26.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2c8bcced-80c2-48c2-bc9a-2590e36ec6e3_calibrate_947246_26.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33cdec5157f7be836bad4c64db5a5e8d with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b149c64f-606f-4f91-91a3-4338e702b416_calibrate_947244_9.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b149c64f-606f-4f91-91a3-4338e702b416_calibrate_947244_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:33 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:34 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fdf6cdf2484ae3c602ec9833c84de9e with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c4e3c8fc-4d9e-4467-b5e1-a876b6af0bb2_calibrate_947140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c4e3c8fc-4d9e-4467-b5e1-a876b6af0bb2_calibrate_947140_28.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a975e8aae63573627d549dc7ba15ef1f with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash a3b3871d08e9aa27df17db18efb01465 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/42b6595c-81ce-421b-954e-8cec390cba3c_calibrate_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/42b6595c-81ce-421b-954e-8cec390cba3c_calibrate_947246_32.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d34b0541-7afc-48b2-a133-a358165ea376_characterizeImage_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d34b0541-7afc-48b2-a133-a358165ea376_characterizeImage_947246_32.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db8e6040ae60b1db07a7b98a277d0e22 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 24ff5bc12c117d09a15464bd22e3fe34 parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ff7e9f34-efc2-4a34-820d-2658fa15ed7f_calibrate_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ff7e9f34-efc2-4a34-820d-2658fa15ed7f_calibrate_947140_40.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c86ab917-1186-4849-81a6-7309a1ed6191_characterizeImage_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/c86ab917-1186-4849-81a6-7309a1ed6191_characterizeImage_947140_40.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0b3b580623d7afd22d41a2761a7165 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 9f52793c7bc3a4ed02754b32b3fac740 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9f69d391-d0c1-49a6-8546-6043e274808b_calibrate_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9f69d391-d0c1-49a6-8546-6043e274808b_calibrate_947142_52.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9a015c11-9484-47dc-b800-b08ef12737af_characterizeImage_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9a015c11-9484-47dc-b800-b08ef12737af_characterizeImage_947142_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:34 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:34 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d6196887ab180746bed53c042af5838 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/42a28457-44ae-4c48-a33d-74c139343c55_calibrate_947142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/42a28457-44ae-4c48-a33d-74c139343c55_calibrate_947142_39.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a030a467260a213a5d82a9fdbdf500 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 99e5f8e08192eb6522425e7f7aff29fb parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0a745fc3-bc8b-4db2-b776-ef1a2c2c634d_calibrate_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0a745fc3-bc8b-4db2-b776-ef1a2c2c634d_calibrate_947248_13.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bda16147-9694-40f2-98a7-ef23850f42f6_characterizeImage_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/bda16147-9694-40f2-98a7-ef23850f42f6_characterizeImage_947248_13.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98db006e9be6bb8987a1b53eebb208de with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5d3de7bd-44a4-478d-b049-5a0596209f2d_calibrate_947246_46.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5d3de7bd-44a4-478d-b049-5a0596209f2d_calibrate_947246_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:35 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 876b5f124cd82df98aaaf2742bca3da8 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 8c0715e23012d55a55e150a00b51ff02 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/330ba905-3755-4fcf-ac64-ca150896e7d8_characterizeImage_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/330ba905-3755-4fcf-ac64-ca150896e7d8_characterizeImage_947248_31.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d78365b56b2c413451168c3a595ff3d1 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d343eec3-0b01-4b8c-b08c-7bd17798e673_calibrate_947140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d343eec3-0b01-4b8c-b08c-7bd17798e673_calibrate_947140_17.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3adf078ce24a22166e51951a2d4b15f with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash aae9e8371e85ef20b64fab4a7197c794 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/71740075-a2dc-48ab-94cd-79c5103dcf23_calibrate_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/71740075-a2dc-48ab-94cd-79c5103dcf23_calibrate_947144_9.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b065a60f-5939-461d-bdbf-2fea561316a6_characterizeImage_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/b065a60f-5939-461d-bdbf-2fea561316a6_characterizeImage_947144_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:35 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75ea0398cea17943f91b0fd7c2564888 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5f38b036-af56-4df1-9d5e-f318d0ff8f86_calibrate_947244_21.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5f38b036-af56-4df1-9d5e-f318d0ff8f86_calibrate_947244_21.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b88d7a9bf35ab1335cc839217adfb2 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5eb212c1-2c5c-4c35-8b73-a643105fa387_calibrate_947244_57.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5eb212c1-2c5c-4c35-8b73-a643105fa387_calibrate_947244_57.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c8bc26261cdfe812ac32c5e3ef9f2bd with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/462639d4-0e2d-4267-94f9-f4892cbacf65_calibrate_947142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/462639d4-0e2d-4267-94f9-f4892cbacf65_calibrate_947142_23.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 268ec7ab6202715edffc4d47b53d5e76 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2f71b868-77db-47a7-993d-caaefa8e2d8c_calibrate_947248_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2f71b868-77db-47a7-993d-caaefa8e2d8c_calibrate_947248_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:36 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a0618a36df55270e9af2f1528934b7 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash dea49fbc1f90ce1130c033e394f121cd parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/45def7db-fc4d-4121-a062-233cd9224108_calibrate_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/45def7db-fc4d-4121-a062-233cd9224108_calibrate_947144_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/dff21387-b017-473a-aced-975060b4f2f9_characterizeImage_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/dff21387-b017-473a-aced-975060b4f2f9_characterizeImage_947144_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:36 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b0f115efb9c9205356e460c68808d74 with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/11a81df3-4307-4813-ace9-a81687309885_calibrate_947142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/11a81df3-4307-4813-ace9-a81687309885_calibrate_947142_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:37 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f5fe2c968e67f67659cc951dadebd6 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash df3b341f438b9d2232f2d4d26496aec3 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/994d60c9-4b22-4b80-a47b-78f823f93368_calibrate_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/994d60c9-4b22-4b80-a47b-78f823f93368_calibrate_947244_54.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eef7741e-d433-46b8-a432-711cde825175_characterizeImage_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/eef7741e-d433-46b8-a432-711cde825175_characterizeImage_947244_54.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7299f76e62716f68743fd796440cb107 with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 02467edc543455c668924a3df2bcb256 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d8d9c8a8-5ddd-406c-af5a-7dc4fc5a93ea_calibrate_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d8d9c8a8-5ddd-406c-af5a-7dc4fc5a93ea_calibrate_947140_44.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/305d708b-19bc-4622-a77e-638b07a9d320_characterizeImage_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/305d708b-19bc-4622-a77e-638b07a9d320_characterizeImage_947140_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:37 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry accf7389549e22c0a73975f372bfd76e with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/412b4515-4180-4f7b-8044-eb836f50c77e_calibrate_947246_34.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/412b4515-4180-4f7b-8044-eb836f50c77e_calibrate_947246_34.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea61d2aa49fdb4fc8694472cb4677d6 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5c4e5e78-b626-4c60-a9ec-3fb44d7f9e2a_calibrate_947246_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/5c4e5e78-b626-4c60-a9ec-3fb44d7f9e2a_calibrate_947246_9.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5887cf56777e81da6dd24bfbf51f8731 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 4b8fb32cb84d0fab56fa62ea7cd0bc39 parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cf585163-4ec2-43bb-ae17-f40d358bc177_calibrate_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cf585163-4ec2-43bb-ae17-f40d358bc177_calibrate_947142_16.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bbb8763d-5f37-4e60-b07b-7a0cb52b3649_characterizeImage_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bbb8763d-5f37-4e60-b07b-7a0cb52b3649_characterizeImage_947142_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:37 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9adccf057b3f7f08a2bf9f3d71003bd with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a57dd677-e3aa-4092-b6df-1cd883244cbd_calibrate_947244_39.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a57dd677-e3aa-4092-b6df-1cd883244cbd_calibrate_947244_39.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0c94d8638d400ceb27127b467986b6d with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a98e32a8-6251-410b-a8ab-d7c6b984e016_calibrate_947244_23.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/a98e32a8-6251-410b-a8ab-d7c6b984e016_calibrate_947244_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:38 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd896bd53b40563786fcffafa1f9f223 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a4acac06-671c-4ed1-b06a-fa728e66508c_calibrate_947142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a4acac06-671c-4ed1-b06a-fa728e66508c_calibrate_947142_3.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09143dd1fbac01594e0a263c72f0f43 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b15aee3d-93de-49c7-97b1-258890adfd63_calibrate_947140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b15aee3d-93de-49c7-97b1-258890adfd63_calibrate_947140_46.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96748ef4287aeca3a79f38cec7a889c8 with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash a2e761bb588380b9eb8016796f66faeb parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/51c97826-623e-4301-98a9-2ed358579b2f_calibrate_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/51c97826-623e-4301-98a9-2ed358579b2f_calibrate_947144_48.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/829c01cf-883e-4a51-af3f-953e2f93ef1b_characterizeImage_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/829c01cf-883e-4a51-af3f-953e2f93ef1b_characterizeImage_947144_48.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b3f9a70acc6983f0dec054d601a9bd with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/20ad85c2-4949-4197-8472-13312739b264_calibrate_947244_35.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/20ad85c2-4949-4197-8472-13312739b264_calibrate_947244_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:38 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba226a8a3269bf9f51bc25f6ef49abb3 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash b363b238c787787ac7bd2410a156ff46 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/de0caadc-ade0-4ffe-b44f-5d4b4c247e75_calibrate_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/de0caadc-ade0-4ffe-b44f-5d4b4c247e75_calibrate_947244_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e0289bf6-21b5-4eba-8c19-98b394706d11_characterizeImage_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e0289bf6-21b5-4eba-8c19-98b394706d11_characterizeImage_947244_40.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7ae0c6c89b266e30c3e2b44172fb569 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 0f1c493dd0206d7632117106430127a6 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8669e552-1088-4e89-89b0-c1914f2a0425_calibrate_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8669e552-1088-4e89-89b0-c1914f2a0425_calibrate_947144_45.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/bc1cad8e-81c3-4df7-8f9b-354a48532fac_characterizeImage_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/bc1cad8e-81c3-4df7-8f9b-354a48532fac_characterizeImage_947144_45.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a80c739cfd5a44c88052b9416627ab2 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4b088098-a2d0-4f43-8708-1bff75550091_calibrate_947144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4b088098-a2d0-4f43-8708-1bff75550091_calibrate_947144_40.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83101dc5f0e188417bc354e60ab6f865 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 5a4699d6e8a68438f6efa13c6ef02dda parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/120286a0-9f22-4a86-8f78-166d39d39582_calibrate_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/120286a0-9f22-4a86-8f78-166d39d39582_calibrate_947144_30.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/c2775a9c-05b3-4f7e-82e0-28b99f1e6e1f_characterizeImage_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/c2775a9c-05b3-4f7e-82e0-28b99f1e6e1f_characterizeImage_947144_30.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9804de8c14110d669ecc8139fcfd3c8 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/95aaf963-039b-4e90-b973-dddfa6049190_calibrate_947140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/95aaf963-039b-4e90-b973-dddfa6049190_calibrate_947140_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:39 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae1d556bb214d1ad169032d562691d35 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash b67677967d369a2c9b8a006ab456303b parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8f2b7306-9527-4a3f-9259-459ca4ea7ba7_calibrate_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8f2b7306-9527-4a3f-9259-459ca4ea7ba7_calibrate_947244_32.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/48c5cbfa-314d-4ed8-9746-2410273bbf59_characterizeImage_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/48c5cbfa-314d-4ed8-9746-2410273bbf59_characterizeImage_947244_32.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42190a9730f46c857f9e0e9a2f666d31 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 091b6cb152609fa8065a83a5dad339d0 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d81daa1e-f3c0-41e1-ab3b-d93ef0e6c6bd_calibrate_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d81daa1e-f3c0-41e1-ab3b-d93ef0e6c6bd_calibrate_947140_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19eb03d1-d253-47c0-b69a-c1038ac1032c_characterizeImage_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/19eb03d1-d253-47c0-b69a-c1038ac1032c_characterizeImage_947140_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:39 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16745e9faa387667387255c46baad3b with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash d45cf573447b6544e7b46831e4ab6d9c parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e14b0cf1-6a3a-4de3-bbd0-1ca30befc1fa_calibrate_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e14b0cf1-6a3a-4de3-bbd0-1ca30befc1fa_calibrate_947142_50.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/32e54e32-4145-4347-a563-73640303cf98_characterizeImage_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/32e54e32-4145-4347-a563-73640303cf98_characterizeImage_947142_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:39 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:39 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0b4a22d4b4e316234558193def17c7 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3bde598b-f5ac-4df3-958a-4526bd10e38c_calibrate_947142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3bde598b-f5ac-4df3-958a-4526bd10e38c_calibrate_947142_26.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eca1e3035417e7300905b93079f8e52 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash ed7ff1f665391a73598479e7af4d083b parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c18dd84d-34c7-4cdd-a904-aeac876dd66c_calibrate_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c18dd84d-34c7-4cdd-a904-aeac876dd66c_calibrate_947246_53.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2c189336-1fdb-447f-b32a-58d8b6b5b3cb_characterizeImage_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/2c189336-1fdb-447f-b32a-58d8b6b5b3cb_characterizeImage_947246_53.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ea84bd96ae6eb664a993bc6ea25782a with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 52907772eef2d5675c29049bedbe7dc3 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f6358306-d4c9-4a58-b6f0-fee41aa1122b_calibrate_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f6358306-d4c9-4a58-b6f0-fee41aa1122b_calibrate_947248_57.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a4cec123-8613-4659-85f5-c65fefc3f6a1_characterizeImage_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/a4cec123-8613-4659-85f5-c65fefc3f6a1_characterizeImage_947248_57.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c5c9c1751c370d5963a4ee4aed420cf with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/4faaeebc-59ed-491b-921a-e1442b73f125_calibrate_947140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/4faaeebc-59ed-491b-921a-e1442b73f125_calibrate_947140_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9265b8755fde1080a652659935443f6b with result from task 803 parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c0fec11-9702-4554-bfd9-d09911cf8103_calibrate_947140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c0fec11-9702-4554-bfd9-d09911cf8103_calibrate_947140_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43cb8537feaf95bb479a4db3bc0d8b26 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 14b11247923b284874bdb3be8217107a parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a02d8552-eab5-4b93-87d0-70e99eec3c50_calibrate_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a02d8552-eab5-4b93-87d0-70e99eec3c50_calibrate_947142_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b611a4ce-6684-413b-8e0c-f9f9033f36f1_characterizeImage_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/b611a4ce-6684-413b-8e0c-f9f9033f36f1_characterizeImage_947142_38.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f20f22b8cb8ddf62124354a26c633d5 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash f1073b1ce65ec97af99a4fab9a3b2199 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/cbdf93b2-1e44-4b9f-b2e0-b523fd8e3a70_calibrate_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/cbdf93b2-1e44-4b9f-b2e0-b523fd8e3a70_calibrate_947244_17.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/a6e93949-2718-4b9f-a300-cd41fc1da6b6_characterizeImage_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/a6e93949-2718-4b9f-a300-cd41fc1da6b6_characterizeImage_947244_17.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b98df980563fcfbc93dcbd4d8293ee9 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9ce3aa21-4ecb-4a2c-82b6-b7845add6d41_calibrate_947144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/9ce3aa21-4ecb-4a2c-82b6-b7845add6d41_calibrate_947144_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:40 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 270c499e72e8cdfac04f6f1a4dbeef8f with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1785eb67-1434-4a1b-b9c5-726f181690fc_calibrate_947144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/1785eb67-1434-4a1b-b9c5-726f181690fc_calibrate_947144_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31652911b1243ce3f4b84169a6356431 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ee02da3d-1662-4fde-966d-005fd936aee2_calibrate_947246_21.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ee02da3d-1662-4fde-966d-005fd936aee2_calibrate_947246_21.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa4e05a8f4217ff908c47d8f5562dd9 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/04a07320-37f5-4f9e-86f4-5adca7402f4b_calibrate_947246_36.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/04a07320-37f5-4f9e-86f4-5adca7402f4b_calibrate_947246_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4414f5ae04031078f039c3540194ae7 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 5b45b1e9e785c7a2cdec95912a0dd545 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3933210d-554d-4fc7-9fdf-12639f9d59ec_calibrate_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3933210d-554d-4fc7-9fdf-12639f9d59ec_calibrate_947244_27.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/883c75a7-f2e8-4ebd-afb3-6313be83ba88_characterizeImage_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/883c75a7-f2e8-4ebd-afb3-6313be83ba88_characterizeImage_947244_27.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68fe2ac6ddb7a4de26ceb772dcf64e70 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/63551d04-db9e-4784-9a26-2aea4d0499f8_calibrate_947140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/63551d04-db9e-4784-9a26-2aea4d0499f8_calibrate_947140_1.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154ed127983f4dbf62787582d5bca398 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 9330b63ad1feaa1af764141106898e12 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c748bf09-0994-42ba-9ee5-5a58b6a614ef_calibrate_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c748bf09-0994-42ba-9ee5-5a58b6a614ef_calibrate_947246_59.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f46b2ea1-1dc9-481f-bdfc-518f39e93176_characterizeImage_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f46b2ea1-1dc9-481f-bdfc-518f39e93176_characterizeImage_947246_59.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e517f4910dd4f4fa8c9c042e781f3d81 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/582412ca-6feb-4590-8d46-2d3a53fd1b58_calibrate_947140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/582412ca-6feb-4590-8d46-2d3a53fd1b58_calibrate_947140_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86ef4c693a080f5f08b770d9eadf6673 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash c28a89b5f71df0c1bcd677f13c364513 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5aa85f43-4d29-43ac-8e7e-5e81de6e71a7_calibrate_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5aa85f43-4d29-43ac-8e7e-5e81de6e71a7_calibrate_947244_3.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c87a454-2385-42b3-8b62-74271ca4bb2d_characterizeImage_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c87a454-2385-42b3-8b62-74271ca4bb2d_characterizeImage_947244_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:41 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86821a3b104098cb7a5666332e1a46f2 with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/b69c0831-9905-43f3-9387-c30520bb7b0b_calibrate_947140_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 291b922fcad1625995cd21031ddd53cb with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c7eae140-d3c0-4473-ab09-ec5ca328f99f_calibrate_947142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c7eae140-d3c0-4473-ab09-ec5ca328f99f_calibrate_947142_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ae2efff0380516fa6c5fcbc6d7534fe with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/9bebec98-c87d-422e-b06f-b00293d4eb0b_calibrate_947244_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/9bebec98-c87d-422e-b06f-b00293d4eb0b_calibrate_947244_8.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e7092410d6cad15601bfbcf4716342c with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash e35731b3a3c53c77045ed1a7d0f407ce parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a021663c-9362-4cf5-bc8c-e9172d867052_calibrate_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a021663c-9362-4cf5-bc8c-e9172d867052_calibrate_947246_38.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/42fdefcb-c66e-4358-ae4f-537f1cae7bb6_characterizeImage_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/42fdefcb-c66e-4358-ae4f-537f1cae7bb6_characterizeImage_947246_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efd0459551c0a31b42739da3c286fb7b with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/466a90e5-6962-4084-8b75-57d9fe02a574_calibrate_947248_37.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/466a90e5-6962-4084-8b75-57d9fe02a574_calibrate_947248_37.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08810dae0b1f40792c83ab9033b1cacc with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/096055ee-8724-41b2-8b71-0de685f39dc7_calibrate_947140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/096055ee-8724-41b2-8b71-0de685f39dc7_calibrate_947140_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:42 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c135b7d9ae6f4fc8d76e6b3415d630c with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/deadd269-99ed-4817-9a8d-9df540328014_calibrate_947246_13.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/deadd269-99ed-4817-9a8d-9df540328014_calibrate_947246_13.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c99881b176188437d4a993367f2e38 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash 69de4b4dfb4ac0a4a177a625168dcbd6 parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1a851e7d-7ccc-4df5-ba71-b42ad7502473_calibrate_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1a851e7d-7ccc-4df5-ba71-b42ad7502473_calibrate_947142_54.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a934af7e-f0e1-4037-ada1-1419e041902d_characterizeImage_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a934af7e-f0e1-4037-ada1-1419e041902d_characterizeImage_947142_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91db10fcd3ee18435114154915fdd078 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a497f62a-94c1-4848-85c4-5b5d86cee23f_calibrate_947144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a497f62a-94c1-4848-85c4-5b5d86cee23f_calibrate_947144_8.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c1d1eb06cf2aaeaed94c64729ad64dd with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash cc456ed158a209fedf687087e8c6a7be parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/998a0992-9973-4d1d-9c5c-5c27ef996376_calibrate_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/998a0992-9973-4d1d-9c5c-5c27ef996376_calibrate_947248_46.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9a3e8239-f487-41b1-b90e-6d90c10ded10_characterizeImage_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9a3e8239-f487-41b1-b90e-6d90c10ded10_characterizeImage_947248_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9425baa0a6604febed3b3973820a1724 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cb3d5d68-435e-427b-924a-1cd9d898ec1a_calibrate_947248_45.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cb3d5d68-435e-427b-924a-1cd9d898ec1a_calibrate_947248_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2dd8871c7333ac08865771a94cc0cb4 with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/6aa28346-f67a-4f63-acc9-3374d1713b7e_calibrate_947144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/6aa28346-f67a-4f63-acc9-3374d1713b7e_calibrate_947144_57.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1998e077258ce65b6cb428768658bdfb with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 3104cb6db8c1a7cf467b196abc5f75c8 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c225ef23-81f9-4d70-8e4d-f0282528e9c0_calibrate_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c225ef23-81f9-4d70-8e4d-f0282528e9c0_calibrate_947140_58.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8af5e340-27af-41b6-8b81-53a409398981_characterizeImage_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8af5e340-27af-41b6-8b81-53a409398981_characterizeImage_947140_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:43 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15e78ec67444cd32bc1772192c7a5746 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3e53fd32-97c9-42bf-9a1c-321629409d34_calibrate_947246_39.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3e53fd32-97c9-42bf-9a1c-321629409d34_calibrate_947246_39.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a71733e6d59445e05f56f81629ecaf7 with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 7a51eb4d97dc418407fd6b1f8c7d58ef parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/dfb055f4-3e22-457d-b03a-a18b7f0f07e5_calibrate_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/dfb055f4-3e22-457d-b03a-a18b7f0f07e5_calibrate_947140_56.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/1790380c-0ec5-47cd-890b-31cc3dad5416_characterizeImage_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/1790380c-0ec5-47cd-890b-31cc3dad5416_characterizeImage_947140_56.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b389ee47ff311805733ba575e36cc8ff with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b301575d-1323-4122-a324-7090899c4832_calibrate_947248_17.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b301575d-1323-4122-a324-7090899c4832_calibrate_947248_17.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90b7b17256ae31b488a88e3f9b7fcd81 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/f43147d1-e611-4aae-af08-5188369f8414_calibrate_947140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/f43147d1-e611-4aae-af08-5188369f8414_calibrate_947140_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:44 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:44 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3221235df861a407080930155b0893 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash d3b90552d3d9492ed2f853cac9db3737 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ca047a3c-890d-470d-9ec5-7f0435d152a7_calibrate_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ca047a3c-890d-470d-9ec5-7f0435d152a7_calibrate_947248_34.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2b10af11-c35b-4411-8ceb-ccb0e6fb9df5_characterizeImage_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/2b10af11-c35b-4411-8ceb-ccb0e6fb9df5_characterizeImage_947248_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:44 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:44 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:44 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72b6ce15c67f8f17e714553fa206b39f with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7eb33e41-eb69-4777-81a8-9fd67999a4f5_calibrate_947142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7eb33e41-eb69-4777-81a8-9fd67999a4f5_calibrate_947142_62.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea4f60f212bb093aa9dd552e66efbcc5 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/4c4112a9-9e30-4a99-8057-1f3b5a6a1a39_calibrate_947142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/4c4112a9-9e30-4a99-8057-1f3b5a6a1a39_calibrate_947142_60.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03367eb0b4bf47062575ab8e749fcaf6 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/effb0ac7-c82b-4723-9209-4f457c285484_calibrate_947142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/effb0ac7-c82b-4723-9209-4f457c285484_calibrate_947142_20.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e78a0a3205cb972e21fcf3a20d962ec with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 0ce29abe843601cb9e9483f94329f28d parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/dea719e7-7eff-424a-afb1-3a70a6305d39_calibrate_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/dea719e7-7eff-424a-afb1-3a70a6305d39_calibrate_947244_30.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e51a6a3d-c1b7-4b3a-9bdd-0ae504b393c9_characterizeImage_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e51a6a3d-c1b7-4b3a-9bdd-0ae504b393c9_characterizeImage_947244_30.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741030aa0c607d6a3b9b6dcdf9c9db76 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3844e23c-857f-486a-81eb-d2f739e68fbb_calibrate_947244_52.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3844e23c-857f-486a-81eb-d2f739e68fbb_calibrate_947244_52.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 173ded12bc857f4691802ccc942c05b0 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 3e967e61869e718fb3be5f2000a0236f parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e355a4c1-f590-4915-8cfe-773cdcb2faca_calibrate_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e355a4c1-f590-4915-8cfe-773cdcb2faca_calibrate_947248_32.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7ee677c7-c3ab-4c1d-b4b0-d98eccb18eba_characterizeImage_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/7ee677c7-c3ab-4c1d-b4b0-d98eccb18eba_characterizeImage_947248_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd3eea06582399248fcd3c2872601832 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/06b63ba7-f9e0-4e91-981e-203542840dcd_calibrate_947246_3.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/06b63ba7-f9e0-4e91-981e-203542840dcd_calibrate_947246_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eae75f6b41763dbc4716d0fe4e36285 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c983af75-9aa5-4eb6-a455-f7ef3439af48_calibrate_947140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c983af75-9aa5-4eb6-a455-f7ef3439af48_calibrate_947140_21.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5304088d20d5bef418fa7bdfcaf65bb with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash db2635b51e834ff727a788b024660aec parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/41eb3935-8e10-45ea-a87a-aa96c8579991_calibrate_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/41eb3935-8e10-45ea-a87a-aa96c8579991_calibrate_947248_36.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d146cbd1-f4d9-4d69-b193-6511a1d7aa38_characterizeImage_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d146cbd1-f4d9-4d69-b193-6511a1d7aa38_characterizeImage_947248_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:45 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b4ff89723cd2d26df435f41d8dbe8e9 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1c213975-336c-4772-86f2-8d2c30d74ee7_calibrate_947246_11.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1c213975-336c-4772-86f2-8d2c30d74ee7_calibrate_947246_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0d86766f3c23d38761d38edb2156022 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/af23dc0c-4772-4417-ba8f-368bc961a4d4_calibrate_947140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/af23dc0c-4772-4417-ba8f-368bc961a4d4_calibrate_947140_59.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4c77c425c53515642daff093f06d59 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5cc0746b-d469-4a80-a242-bc5cefc97403_calibrate_947142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5cc0746b-d469-4a80-a242-bc5cefc97403_calibrate_947142_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21100d541d71393c51fb633acf371db1 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/a85edb93-443c-4065-92df-c5a4a9c23541_calibrate_947140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/a85edb93-443c-4065-92df-c5a4a9c23541_calibrate_947140_42.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66cdb0a89a48c9022aeb0e2246c9f38f with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 7c333831c28f50bc6d7b444db3195bdc parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ebe573d1-246d-4352-9be1-57fdc6592ba1_calibrate_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ebe573d1-246d-4352-9be1-57fdc6592ba1_calibrate_947140_35.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8cc49a56-c984-49f6-b1f2-231c6b53afd6_characterizeImage_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/8cc49a56-c984-49f6-b1f2-231c6b53afd6_characterizeImage_947140_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4691fd950c9e08f5636121904f416c52 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/120dbe16-6fd5-41a0-83b6-9186f0f7d8c4_calibrate_947246_8.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/120dbe16-6fd5-41a0-83b6-9186f0f7d8c4_calibrate_947246_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e4c9b1fad96a997c983f5100706c6d4 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/6643f9e5-9ab0-4349-9063-45aff6785571_calibrate_947140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/6643f9e5-9ab0-4349-9063-45aff6785571_calibrate_947140_48.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80c0e49b3d780c98429de7a18ddcae3 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 4b4d440775809e19d4bc6dc873243879 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fd151ad1-5541-4a41-8b47-d53ae59ad987_calibrate_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fd151ad1-5541-4a41-8b47-d53ae59ad987_calibrate_947140_54.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/a0f342ee-c2ad-4e20-be77-407c48bfa37c_characterizeImage_947140_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:46 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b625d5c00a249572c1eb4875474e3614 with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 7dcd709f495cdfc1e03816b9ac9a19d3 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/38acd200-0320-4aab-a4f8-75e3c820c6bd_calibrate_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/38acd200-0320-4aab-a4f8-75e3c820c6bd_calibrate_947246_20.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/37222aeb-bb01-43b8-9531-57790485480d_characterizeImage_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/37222aeb-bb01-43b8-9531-57790485480d_characterizeImage_947246_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d0d2b16d263d66e2a8fd7940a07fb6a with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/3b08281a-6c9f-4935-8625-f75662a712a7_calibrate_947142_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6937edff8b0e9bedb957df20b909dfb with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bf39de38-cec8-4129-b864-8d16601903ae_calibrate_947248_55.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bf39de38-cec8-4129-b864-8d16601903ae_calibrate_947248_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:47 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 240 parsl.dataflow.strategy DEBUG: Executor multi has 240 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 240 parsl.dataflow.strategy DEBUG: Executor multi has 240 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9354935f40825340216b3619580c496 with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d1fd6468-2b11-4874-bb0d-ce5f77b8dbed_calibrate_947246_14.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d1fd6468-2b11-4874-bb0d-ce5f77b8dbed_calibrate_947246_14.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73891e9846e4de07dd31e0094cff420c with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/aafa1946-1fd0-4bdb-9d47-244752c6873e_calibrate_947140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/aafa1946-1fd0-4bdb-9d47-244752c6873e_calibrate_947140_5.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f88d76ee41cc5fad2d521e9b9713ef0c with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1e63a0f6-7c0a-4977-ba53-62b5ea0350ac_calibrate_947140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1e63a0f6-7c0a-4977-ba53-62b5ea0350ac_calibrate_947140_18.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e96d0bbee2a1c933b60b3e06df484c81 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f6e4a225-151f-4402-873a-43263d897b8a_calibrate_947144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f6e4a225-151f-4402-873a-43263d897b8a_calibrate_947144_7.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b01f6b1ea1a079b0c4b35865f824ef96 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5d7553dc-5528-4f3d-8668-91e36d07f8b7_calibrate_947142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5d7553dc-5528-4f3d-8668-91e36d07f8b7_calibrate_947142_1.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ed78ab5fa4691791f63e78d07de7a78 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/77808938-4245-42a6-9a06-b5056d11799b_calibrate_947244_45.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/77808938-4245-42a6-9a06-b5056d11799b_calibrate_947244_45.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9905ad05596b7aa0f51e30f980284a7d with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/51a19127-29c4-40f1-9305-cd8c5bc1cbad_calibrate_947140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/51a19127-29c4-40f1-9305-cd8c5bc1cbad_calibrate_947140_11.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87dde45bfdcab50e7ca6646d41512a2f with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash cf61c570d91143b17e1d4bec6de11fae parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/829f4265-b8e5-4264-b97c-a7e02f35615a_calibrate_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/829f4265-b8e5-4264-b97c-a7e02f35615a_calibrate_947246_24.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5602e84e-e334-44b4-9270-4f0295ea31fd_characterizeImage_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/5602e84e-e334-44b4-9270-4f0295ea31fd_characterizeImage_947246_24.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a86acb109b8163ecaa54bdbc59c3bed6 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/38355f3e-582c-49fb-8b9d-43d5d4f4c365_calibrate_947144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/38355f3e-582c-49fb-8b9d-43d5d4f4c365_calibrate_947144_34.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cafe10fcb292d3017de4672221b7e2d1 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 83a06a13241fdf19ecf8945b20bf6f7a parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/986aeea4-f053-4e83-ad18-133fda3d12a6_calibrate_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/986aeea4-f053-4e83-ad18-133fda3d12a6_calibrate_947244_13.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9a9efe67-2781-4b7c-a3d0-8109e1e3dc74_characterizeImage_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/9a9efe67-2781-4b7c-a3d0-8109e1e3dc74_characterizeImage_947244_13.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5a8d25ce260e578152a39e07bddf52 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 131e0570f61425a23a7b90a6fdde42ef parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c035ae5f-5590-41b0-b2ab-dfe0b0728d7a_calibrate_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c035ae5f-5590-41b0-b2ab-dfe0b0728d7a_calibrate_947248_6.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3214bbbd-21b8-40ca-9416-bfd77162ddf8_characterizeImage_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3214bbbd-21b8-40ca-9416-bfd77162ddf8_characterizeImage_947248_6.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e9cd3ddb4a6db8be0b8cf82f1e0878f with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 154b5ecb7ef8d47be756b148e0e08968 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69234c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/91bc6497-f2dd-4df0-a4a5-1585c804e21a_calibrate_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/91bc6497-f2dd-4df0-a4a5-1585c804e21a_calibrate_947248_19.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f147fef3-43e8-4a57-808a-d7706b06bf5a_characterizeImage_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f147fef3-43e8-4a57-808a-d7706b06bf5a_characterizeImage_947248_19.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e274537a700cdcfa4fc1e3a9d3b5345c with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1c54eb48-e2af-4278-ac44-0e0182f16fe9_calibrate_947244_47.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1c54eb48-e2af-4278-ac44-0e0182f16fe9_calibrate_947244_47.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dab047863ae903ded10785ed761c629a with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7ade571c-536b-431c-b9e3-ca5e6ee96cfb_calibrate_947142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/7ade571c-536b-431c-b9e3-ca5e6ee96cfb_calibrate_947142_21.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219a7eb65c568e3817a29c63f7680210 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash f7682ee9e6f9baf9febbe94911d06e21 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4dec69b5-4585-4b1e-aea2-cbc6c14f4b82_calibrate_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4dec69b5-4585-4b1e-aea2-cbc6c14f4b82_calibrate_947244_46.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5b292fce-7cca-4191-91d4-2128d947fb91_characterizeImage_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/5b292fce-7cca-4191-91d4-2128d947fb91_characterizeImage_947244_46.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 064da76330d1cf53ee282b56a13e3b07 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash f7cf50c8d30902d12367dbb09a7cabe2 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6976f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8c56c037-feeb-47d9-ab8a-eac095c7b8e5_calibrate_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8c56c037-feeb-47d9-ab8a-eac095c7b8e5_calibrate_947246_50.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/bf64f706-93a4-4e93-a4d1-193c46a9553e_characterizeImage_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/bf64f706-93a4-4e93-a4d1-193c46a9553e_characterizeImage_947246_50.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b95237544f06b72e2e0d38cebd4a487 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/775f8c7d-b49d-4073-93bd-ccf3d5df272b_calibrate_947246_30.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/775f8c7d-b49d-4073-93bd-ccf3d5df272b_calibrate_947246_30.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f702673f8e796bc0526333f37823af with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1857a075-8860-45f1-bdfd-2f33c4a563dd_calibrate_947246_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/1857a075-8860-45f1-bdfd-2f33c4a563dd_calibrate_947246_60.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92c03e0b417a58fec98f7344284807cd with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a9ab60fe-1833-4007-b2b4-b69f16ea0900_calibrate_947246_62.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a9ab60fe-1833-4007-b2b4-b69f16ea0900_calibrate_947246_62.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bdb9f72d2e6930e39bb4d257931893e with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/29da6807-6727-4a32-9b19-a926bf18dcca_calibrate_947144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/29da6807-6727-4a32-9b19-a926bf18dcca_calibrate_947144_47.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f01d9f9c3eb5925003e524648908306 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 4cf178cefdede39dcf3880e8f7559c2f parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/56597cb9-7126-4c1e-ae4c-b67f2d4fbc66_calibrate_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/56597cb9-7126-4c1e-ae4c-b67f2d4fbc66_calibrate_947144_55.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4c104db1-a326-4816-b29d-90627f1809f9_characterizeImage_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/4c104db1-a326-4816-b29d-90627f1809f9_characterizeImage_947144_55.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 655bfe4fcc0dda81cf667504ebf481e9 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash a7043daf8c9aac4ec9e6d4759aa3b83d parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/52d55fdc-a703-445c-86a1-d074f14de7a6_calibrate_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/52d55fdc-a703-445c-86a1-d074f14de7a6_calibrate_947142_53.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcac8c1a-a240-4c18-a779-b9263e35ada5_characterizeImage_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcac8c1a-a240-4c18-a779-b9263e35ada5_characterizeImage_947142_53.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea2cd8cbe33c3b3c4e994c3ce413110 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f7dc9fb6-5bce-4613-ac1e-f0495dc64655_calibrate_947144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/f7dc9fb6-5bce-4613-ac1e-f0495dc64655_calibrate_947144_19.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 974f873bdeaa927146ad7dbe6cafd42d with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c6a8951e-bf4e-46f4-a0ed-d0cfea21098d_calibrate_947140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c6a8951e-bf4e-46f4-a0ed-d0cfea21098d_calibrate_947140_26.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 062fa3537fb08c68e0ba8aa84e49ca9d with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 5f0cc0cc5b51518c8a4b0e9d7eb92af7 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/cc1471e9-70bd-4643-afa1-82438c4e8234_calibrate_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/cc1471e9-70bd-4643-afa1-82438c4e8234_calibrate_947140_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/6cadbf45-8112-4ad5-a663-85c6d72e320c_characterizeImage_947140_9.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af6a7dab2463fbf746aa9b1fc2574d9b with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8aa156ea-9810-4421-88a0-9fe0b2b85188_calibrate_947142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8aa156ea-9810-4421-88a0-9fe0b2b85188_calibrate_947142_43.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef3ce33efc968523168232c2266d186c with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 03f0e96ba119876434d77686460798c4 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/591555c0-26d0-4671-82d4-30d22bffd6b2_calibrate_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/591555c0-26d0-4671-82d4-30d22bffd6b2_calibrate_947244_60.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/2b43e197-7e40-4a56-b074-1bd339e6e6fc_characterizeImage_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/2b43e197-7e40-4a56-b074-1bd339e6e6fc_characterizeImage_947244_60.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0034d1b4fcd82cf0e2bbb51c567e5d23 with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c8d22479-4606-426e-9735-84adea0d76e0_calibrate_947142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/c8d22479-4606-426e-9735-84adea0d76e0_calibrate_947142_32.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry accce9ec7b8f0e9b483672751dc30df6 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash d051109662b8b8896ea669acdb27f818 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/51d3d5fb-81e5-4385-8a2f-2c1562142007_calibrate_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/51d3d5fb-81e5-4385-8a2f-2c1562142007_calibrate_947246_4.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/00d612a2-fecc-4e0a-b52f-0fae14f8ab60_characterizeImage_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/00d612a2-fecc-4e0a-b52f-0fae14f8ab60_characterizeImage_947246_4.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7eec916d3cf0b2a6f67dff9c39f657a with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 00dd44d82df430ed76cb67ea5d0bd7a2 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d336f693-7add-4be1-9b05-7db32542213d_calibrate_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d336f693-7add-4be1-9b05-7db32542213d_calibrate_947244_50.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/108b697b-ecd0-472d-a44f-a8d7ba4a5798_characterizeImage_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/108b697b-ecd0-472d-a44f-a8d7ba4a5798_characterizeImage_947244_50.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 222 parsl.dataflow.strategy DEBUG: Executor multi has 222 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b20c9056f530959fa72841f2b46004 with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1a531107-4abe-4f45-a4dd-d9b9d0a4f90c_calibrate_947140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/1a531107-4abe-4f45-a4dd-d9b9d0a4f90c_calibrate_947140_55.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce4c834fee439d6680ca5928587d3d48 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c1a2f34-a22c-406d-be9a-f21d80e9b358_calibrate_947140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/3c1a2f34-a22c-406d-be9a-f21d80e9b358_calibrate_947140_36.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa1ec056c8a4a72f9c062e339e8234d with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 6aa7306644f7ffd6aaa38610da94aaf7 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bc834afc-cc7d-4487-b811-4128e1a91c44_calibrate_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bc834afc-cc7d-4487-b811-4128e1a91c44_calibrate_947248_24.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8ffb6ad5-3d42-450e-b824-8666af694bbf_characterizeImage_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8ffb6ad5-3d42-450e-b824-8666af694bbf_characterizeImage_947248_24.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06f47bd5e15f963f54dc6a6a2ee8b835 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 563fad703e0f85e9f902cb4941909d47 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/72b43d55-4d0e-4492-998f-9cf01ede155c_calibrate_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/72b43d55-4d0e-4492-998f-9cf01ede155c_calibrate_947244_42.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a0f4434-c7bd-47a0-a2b5-273b7796f344_characterizeImage_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1a0f4434-c7bd-47a0-a2b5-273b7796f344_characterizeImage_947244_42.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46c749973ebde635fd49fc4b2676272c with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 1f2117f2c63494c448dc0872cd2682d1 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41a80556-6597-41b8-8925-744eea1fb15f_calibrate_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41a80556-6597-41b8-8925-744eea1fb15f_calibrate_947144_1.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3968f56b-3311-4e58-972a-42a14058c4d8_characterizeImage_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/3968f56b-3311-4e58-972a-42a14058c4d8_characterizeImage_947144_1.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d3874dd1e84935c0edee623e521bcfc with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03705dfa-a993-4cfe-9ace-0be4b14588f5_calibrate_947248_43.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03705dfa-a993-4cfe-9ace-0be4b14588f5_calibrate_947248_43.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e265f4a6f40d6f0f2150b92a37faa6 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash d77a7567b1df7cf4a89961859f0bd0d8 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf694c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a7d14516-90db-4aac-80c6-4c5161ed4b87_calibrate_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a7d14516-90db-4aac-80c6-4c5161ed4b87_calibrate_947144_29.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e80e5349-99c0-43c6-a43d-61a83ea423ee_characterizeImage_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e80e5349-99c0-43c6-a43d-61a83ea423ee_characterizeImage_947144_29.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65cf9f0b50e079f5265a3339d2cd1429 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a3e8f0b2-5402-4121-94e2-bf7189328f13_calibrate_947144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a3e8f0b2-5402-4121-94e2-bf7189328f13_calibrate_947144_32.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83bc95206ae25890fee6e5bbdd62fde3 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 2fa16f2a8b3457a6c0c6cf2ac4d403ce parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7a3d6cd7-aed0-4f97-b412-8cba5196799c_calibrate_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7a3d6cd7-aed0-4f97-b412-8cba5196799c_calibrate_947244_11.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/c4cbe9b5-eaed-4d69-9dd8-19a287fb8eb4_characterizeImage_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/c4cbe9b5-eaed-4d69-9dd8-19a287fb8eb4_characterizeImage_947244_11.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718bd08871d0f9d3d8edac478616ea9d with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d606e417-3269-48da-a33d-b94501c06748_calibrate_947246_58.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d606e417-3269-48da-a33d-b94501c06748_calibrate_947246_58.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7230a1fc8b00d3c9cda5588895b285a1 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7115362f-5706-4b03-8802-63751fb03b65_calibrate_947248_3.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7115362f-5706-4b03-8802-63751fb03b65_calibrate_947248_3.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c6277763320ef3ac3b2f72960c1b901 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 1a1c12ed045b14e9de9a86e9613ea0a6 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4a865334-ca9b-4cdb-8020-b9537c1d4dc3_calibrate_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4a865334-ca9b-4cdb-8020-b9537c1d4dc3_calibrate_947244_7.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/6c38d1d7-a977-4531-b019-9f9408b742df_characterizeImage_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/6c38d1d7-a977-4531-b019-9f9408b742df_characterizeImage_947244_7.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 788b423ac851b9fc6a5a080bd9a8f923 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/575a5b63-2ee9-45a0-89ed-dfcfa9c9a9b8_calibrate_947142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/575a5b63-2ee9-45a0-89ed-dfcfa9c9a9b8_calibrate_947142_42.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b07679e01b7f3f8b103a004d28d96b04 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2efc6364-f06c-4fef-90c7-d6381cd33b12_calibrate_947144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2efc6364-f06c-4fef-90c7-d6381cd33b12_calibrate_947144_10.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry babb9ffe214a584941c84513185883a4 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8e9eed70-98c3-4719-8ebc-c575b76e6a27_calibrate_947140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/8e9eed70-98c3-4719-8ebc-c575b76e6a27_calibrate_947140_24.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15d5377db0d06ccb9ee4ef4cf402c00 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a9c7c657-733d-48cb-ad33-0c49e50eb963_calibrate_947144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a9c7c657-733d-48cb-ad33-0c49e50eb963_calibrate_947144_59.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0444271f46432f864fa6329d877ec5cc with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/78c5c431-ad84-431f-8a4f-0e6260b2c4ef_calibrate_947248_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/78c5c431-ad84-431f-8a4f-0e6260b2c4ef_calibrate_947248_56.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d24e32a13609c91ba99d7d4b5ff687f6 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/456e2735-9986-44c2-b997-b8dfaa32c48f_calibrate_947144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/456e2735-9986-44c2-b997-b8dfaa32c48f_calibrate_947144_24.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcea83eac7fa18950070a3902418680d with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dd949347-2f7e-4e06-8713-cb7819c9a359_calibrate_947144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/dd949347-2f7e-4e06-8713-cb7819c9a359_calibrate_947144_3.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd0fe128166447f688a87515e0d5a7ed with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 12408b6f50a0032f0ad690d393aaadba parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bbe367db-b29a-4ac8-8b4e-9597bd7a03a0_calibrate_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bbe367db-b29a-4ac8-8b4e-9597bd7a03a0_calibrate_947244_20.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/041adb41-1524-409c-b26f-43c927c842e7_characterizeImage_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/041adb41-1524-409c-b26f-43c927c842e7_characterizeImage_947244_20.stderr 2024-03-14 05:37:57 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:37:57 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:57 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 263f0c123edddaddbfa7094945998a5d with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/539f94ee-136d-43f4-8acb-9f4e98d13b45_calibrate_947248_44.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/539f94ee-136d-43f4-8acb-9f4e98d13b45_calibrate_947248_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:58 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abe61c0817546d4d5ec8348dfe47c914 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash e27fcb251f366dc7284ed7139a065016 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7e7f8eaf-bfda-4ffb-9a4c-bcc68bfe1b64_calibrate_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7e7f8eaf-bfda-4ffb-9a4c-bcc68bfe1b64_calibrate_947248_35.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4b42e88d-b0d5-4009-ab4d-919b423f9e7b_characterizeImage_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/4b42e88d-b0d5-4009-ab4d-919b423f9e7b_characterizeImage_947248_35.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbbf13947059badee0825443842af258 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c1d9cd63-205d-46c5-9db3-c827a75d8d2e_calibrate_947248_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 652938195e0269f9fe25c9a101453fc9 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0f0c2d45-ff2d-4ecc-ad61-2a057f194960_calibrate_947244_34.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e887ddb4f423ceb190d59846d9a0d32 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/57966154-127f-45da-b757-88ebeba118fa_calibrate_947244_24.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/57966154-127f-45da-b757-88ebeba118fa_calibrate_947244_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 497 try 0 failed parsl.dataflow.dflow INFO: Task 497 marked for retry parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 8c0715e23012d55a55e150a00b51ff02 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6975c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 1 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1d228b1361d34607340d4dca1e7fe0a with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c728c013-56de-4f15-b09d-edf704c4316a_calibrate_947140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c728c013-56de-4f15-b09d-edf704c4316a_calibrate_947140_22.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3722416e1ef352240b818e5ce970f967 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 210303bdf4f0690f692d774846f93d6c parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc9744f2-dd2c-43f6-9bf5-55f2fe3f5269_calibrate_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc9744f2-dd2c-43f6-9bf5-55f2fe3f5269_calibrate_947140_34.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0fc41aca-6886-4dd0-9906-18b1d134ad22_characterizeImage_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/0fc41aca-6886-4dd0-9906-18b1d134ad22_characterizeImage_947140_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:37:59 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a72d9ab0d7645801633a3db377d346c with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/17bec250-732a-4283-a22b-97b3ec49688b_calibrate_947142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/17bec250-732a-4283-a22b-97b3ec49688b_calibrate_947142_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df09fbf1e47dbbe7f130f9fa4efc5bb with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 1caef8a9af0410d6ec7b9f4564a29753 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/61eef436-f546-4785-8417-3d7ef1cc8c62_calibrate_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/61eef436-f546-4785-8417-3d7ef1cc8c62_calibrate_947246_23.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f391430e-f814-427c-8ae7-e993f6a757a1_characterizeImage_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f391430e-f814-427c-8ae7-e993f6a757a1_characterizeImage_947246_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39d3b88751220709e4326e092cc7fd15 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash f01b3c9cb099481d5787e68f84ea7d42 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/37eba8b1-2050-4e1d-84b3-69565b130a3b_calibrate_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/37eba8b1-2050-4e1d-84b3-69565b130a3b_calibrate_947142_49.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/aea8f30a-3e00-4921-9653-5030e9b61ba1_characterizeImage_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/aea8f30a-3e00-4921-9653-5030e9b61ba1_characterizeImage_947142_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:00 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dde750cb26fa2a0c5f135809d45c40f with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0fe5128e-3caa-4a29-996e-18db426c4e49_calibrate_947142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/0fe5128e-3caa-4a29-996e-18db426c4e49_calibrate_947142_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4eb5571f9c0b050791c6c32450017d0 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash d30ed791c3878755007cc251ba79f7b4 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6878180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f5efcaa6-592a-468f-a02a-4c9b28fc5882_calibrate_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f5efcaa6-592a-468f-a02a-4c9b28fc5882_calibrate_947248_52.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/809118ae-0d5e-4f3f-ad99-5f437d00ddba_characterizeImage_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/809118ae-0d5e-4f3f-ad99-5f437d00ddba_characterizeImage_947248_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b83d199ed38296680a4da19e53808c6 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3bea1026-48ed-49b2-819b-ec6800b140c1_calibrate_947244_56.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3bea1026-48ed-49b2-819b-ec6800b140c1_calibrate_947244_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a615771cf4cf900a22163d01f66d1cb0 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/db4b072a-9224-4286-90c1-be8899cfd106_calibrate_947144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/db4b072a-9224-4286-90c1-be8899cfd106_calibrate_947144_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4303df5a728b5e45953c4df032072df9 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2d851ad2-2102-4bc1-9ac9-6ccf02eafac4_calibrate_947246_44.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/2d851ad2-2102-4bc1-9ac9-6ccf02eafac4_calibrate_947246_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:01 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a53b48eef9f672eb253fe64071c44389 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/36829a68-6c4b-49f8-8f8a-542262bf0e25_calibrate_947140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/36829a68-6c4b-49f8-8f8a-542262bf0e25_calibrate_947140_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac09530b4e37b672a12fc994f5237ecb with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/79b8fae2-f7ac-40b1-ac03-7635a5e3d06b_calibrate_947248_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0bf887063c9ca8a6f661f00269caac with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash beaf6488041fc13c1d52b7f0730e95d5 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/4ef956d3-365f-4209-9df3-f02e0e8545b0_calibrate_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/4ef956d3-365f-4209-9df3-f02e0e8545b0_calibrate_947248_62.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3f0218bc-d2c6-4052-ad3d-a2db49249086_characterizeImage_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/3f0218bc-d2c6-4052-ad3d-a2db49249086_characterizeImage_947248_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a60bbf082222cc9c6395e75e4ab26cb with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b24da02c-ca23-41ea-81f5-3dfd9db7a9ff_calibrate_947246_19.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b24da02c-ca23-41ea-81f5-3dfd9db7a9ff_calibrate_947246_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:02 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 460ca9782f903957208fc0f9c7544a87 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3dab620-1d03-4cbf-be5f-d9249ebd6bff_calibrate_947246_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3dab620-1d03-4cbf-be5f-d9249ebd6bff_calibrate_947246_15.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ffd7e8c2c6dfcfe956c03dfa4aedc5c with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/69f093bb-e71a-4717-9952-4734a4dd3c81_calibrate_947248_4.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/69f093bb-e71a-4717-9952-4734a4dd3c81_calibrate_947248_4.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d016317b8c51366bf6ccb9bebc2e50 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e81cb99-7ba8-4446-b0f7-47e970394772_calibrate_947142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e81cb99-7ba8-4446-b0f7-47e970394772_calibrate_947142_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae2bc283ba8931ef601e1f56679a925 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2aba5df2-cc01-4b04-bfad-b164d42200ab_calibrate_947144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/2aba5df2-cc01-4b04-bfad-b164d42200ab_calibrate_947144_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1527d3615ef77a7dddda06fd3d5814ae with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 3dc51a86956333cd2f9193e0ef0d8af8 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08ad0592-6147-4c40-a192-fded2e06f0a4_calibrate_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08ad0592-6147-4c40-a192-fded2e06f0a4_calibrate_947248_11.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8f1cb8cf-c7e4-4594-ad46-8cd0d79f95ff_characterizeImage_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8f1cb8cf-c7e4-4594-ad46-8cd0d79f95ff_characterizeImage_947248_11.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f727c9baefb2cb7379999ac7216677f6 with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash accc7a665bb97ab66c69fe3f77c3bf6f parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/740154ec-1a1a-4faf-bdd5-5cddeb58ad53_calibrate_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/740154ec-1a1a-4faf-bdd5-5cddeb58ad53_calibrate_947144_56.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/00d9443d-4c67-4a75-a712-a8e176c91368_characterizeImage_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/00d9443d-4c67-4a75-a712-a8e176c91368_characterizeImage_947144_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77a62315cddf158d989001ac505e97c5 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fda35ca5-2068-4d33-9a0f-18d832bd5d29_calibrate_947140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fda35ca5-2068-4d33-9a0f-18d832bd5d29_calibrate_947140_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:03 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 891f8ee35997c268b06611b74102e416 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3450d34a-2e6b-4c18-8fe1-fef612fddec6_calibrate_947144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3450d34a-2e6b-4c18-8fe1-fef612fddec6_calibrate_947144_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db88015e9e7c661a24aac9a90ca31c5 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41e29fd2-a722-4f58-9998-ac1e4af3262a_calibrate_947144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41e29fd2-a722-4f58-9998-ac1e4af3262a_calibrate_947144_20.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7ebae492482b4c89974115b0e8b44d with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3bb61ef-f9a8-41cf-b4f6-fe226314f0e2_calibrate_947246_40.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c3bb61ef-f9a8-41cf-b4f6-fe226314f0e2_calibrate_947246_40.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dfc2aa30a26397874f5a1caba6ec659 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bad552e7-9e70-4e8f-865e-72bd8ea50813_calibrate_947140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bad552e7-9e70-4e8f-865e-72bd8ea50813_calibrate_947140_12.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa37f494feecfc5a596ba39456a1e6ba with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash c50b491b72369c34d46143ff7bb56daa parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1fea23fa-17e7-4d66-aeb6-b16aef280524_calibrate_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1fea23fa-17e7-4d66-aeb6-b16aef280524_calibrate_947142_44.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/401f1426-b0c3-4e83-9784-e6f44e5560bc_characterizeImage_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/401f1426-b0c3-4e83-9784-e6f44e5560bc_characterizeImage_947142_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c284e3f40ac0e587de6b2a43f9c05681 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash cc2b6cc7325078d46ec0e2de60a7f251 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/89447251-635b-443d-bf21-31ef2ab88589_calibrate_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/89447251-635b-443d-bf21-31ef2ab88589_calibrate_947248_50.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c510443b-f296-4023-b45f-250e0d3c9237_characterizeImage_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c510443b-f296-4023-b45f-250e0d3c9237_characterizeImage_947248_50.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40910a202537a35cda9163e2debb580f with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash fa60c8a845a3677312bafe69b09e9804 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/629a6c57-c4b6-43cf-b0bb-56e3a63d8dc7_calibrate_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/629a6c57-c4b6-43cf-b0bb-56e3a63d8dc7_calibrate_947144_42.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ecff300b-aae7-4929-82dd-a1ca5734d6af_characterizeImage_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ecff300b-aae7-4929-82dd-a1ca5734d6af_characterizeImage_947144_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e89a38cbb5f26255893f2e96e4812350 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/c6dd51a4-69a4-4440-9225-140a7cf0b061_calibrate_947144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/c6dd51a4-69a4-4440-9225-140a7cf0b061_calibrate_947144_58.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6e87e03d4ed6e71a172c8c2c9fd63b with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4c232da1-7d1b-4fea-9ee4-56e5462bebf3_calibrate_947246_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4c232da1-7d1b-4fea-9ee4-56e5462bebf3_calibrate_947246_43.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d027a8d157ce52e0c591d56350548441 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3a9341d3-94f0-4fe7-b1f0-88f9482c95db_calibrate_947244_5.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3a9341d3-94f0-4fe7-b1f0-88f9482c95db_calibrate_947244_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 039076019bd6e36828def09aad960f10 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/677d4273-7fbb-4dc6-a531-518008222cc0_calibrate_947248_15.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/677d4273-7fbb-4dc6-a531-518008222cc0_calibrate_947248_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:04 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b21d7d19d7e5a9fc8b9cfdf461b4b9a8 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e1ce4e2-3116-4656-8725-049df9095e58_calibrate_947142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/8e1ce4e2-3116-4656-8725-049df9095e58_calibrate_947142_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad2e19ba24a793940bff4a2cbecf346 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cee8d462-ee85-4680-b18f-d14d3473d83f_calibrate_947142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cee8d462-ee85-4680-b18f-d14d3473d83f_calibrate_947142_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ff5bc12c117d09a15464bd22e3fe34 with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ff7e9f34-efc2-4a34-820d-2658fa15ed7f_calibrate_947140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ff7e9f34-efc2-4a34-820d-2658fa15ed7f_calibrate_947140_40.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd0b9ff36cb672ece7a8f736bc7e048 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d88bc845-5338-480d-adf5-a3e39a19aa14_calibrate_947144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d88bc845-5338-480d-adf5-a3e39a19aa14_calibrate_947144_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89f16adf3b1b3ba37dfbf35d63cd39e5 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/dbfbcf17-ef52-4329-b9b7-37bc8c39344f_calibrate_947246_10.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/dbfbcf17-ef52-4329-b9b7-37bc8c39344f_calibrate_947246_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471c3c4dcf36fee6f65b5d4d3097c6c0 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash db487e2be836737d82bf154221c4d7ba parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c166304b-0652-4d4f-9772-419b0f09e2e9_calibrate_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c166304b-0652-4d4f-9772-419b0f09e2e9_calibrate_947248_61.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c52c15dd-7eba-4d55-94f3-36eeca94fa83_characterizeImage_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/c52c15dd-7eba-4d55-94f3-36eeca94fa83_characterizeImage_947248_61.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80d55103ac8dd4b09dbec381c13713b2 with result from task 982 parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 84eb2e3e74fd0e9e3e40840fc035ddc8 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/890278a1-1a93-4f47-bedf-3d3c69645429_calibrate_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/890278a1-1a93-4f47-bedf-3d3c69645429_calibrate_947244_41.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ffc6396c-30d3-48fb-96cb-31e7bcb7f9b3_characterizeImage_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/ffc6396c-30d3-48fb-96cb-31e7bcb7f9b3_characterizeImage_947244_41.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c8e4edfe83177e3725ba6367fd9693e with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/fc5dcfb2-eea0-4f39-a903-a72f0e84ebde_calibrate_947248_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:05 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70507ae2b31232ef2532ae173ca85b5a with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/cfd97af0-c520-419c-8d33-2fb87286854d_calibrate_947246_41.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/cfd97af0-c520-419c-8d33-2fb87286854d_calibrate_947246_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:06 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3d93a7362526b434c5ad841760d2786 with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/49bbb636-5b25-4171-b11b-3cbe78ff1bd7_calibrate_947244_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/49bbb636-5b25-4171-b11b-3cbe78ff1bd7_calibrate_947244_53.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec341faa6d143332510aaafb1f460f4 with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 7af165a119e62411e9e37887de62e45a parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ad3e5756-252c-44d5-9672-073b3c8f4375_calibrate_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ad3e5756-252c-44d5-9672-073b3c8f4375_calibrate_947248_9.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d9458d79-5e6c-4ad5-ae17-c3753f204dd8_characterizeImage_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d9458d79-5e6c-4ad5-ae17-c3753f204dd8_characterizeImage_947248_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f82434364db615261dc8f4d36310dba0 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b7250007-43e3-45c5-adc8-9ba9692d7e03_calibrate_947246_7.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b7250007-43e3-45c5-adc8-9ba9692d7e03_calibrate_947246_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5c4791033829c22f7b53207837f7b9c with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash ac41395f7566a7b505002cdccbcec670 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d95c46c9-a77b-4420-bcbf-d3c499041213_calibrate_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d95c46c9-a77b-4420-bcbf-d3c499041213_calibrate_947244_18.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/711e4a7a-a7a0-45e6-b04c-19889170dac2_characterizeImage_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/711e4a7a-a7a0-45e6-b04c-19889170dac2_characterizeImage_947244_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:07 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1510ac395a5df2c487cc03d9ed2fd425 with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash b7cf127fbbc6f6dbd46369359e168faa parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f9cb3895-9919-45ca-9484-1f765d8de649_calibrate_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f9cb3895-9919-45ca-9484-1f765d8de649_calibrate_947142_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a3804d6d-af6d-4c13-bf1a-caad4fa23345_characterizeImage_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/a3804d6d-af6d-4c13-bf1a-caad4fa23345_characterizeImage_947142_29.stderr parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02467edc543455c668924a3df2bcb256 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d8d9c8a8-5ddd-406c-af5a-7dc4fc5a93ea_calibrate_947140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d8d9c8a8-5ddd-406c-af5a-7dc4fc5a93ea_calibrate_947140_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2dfa46e909c52f70fa5b272dc3d2f23 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash c2a1928eb6711811f9bceb8a73b8b819 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3fcd2e9c-bcb6-4e96-a0a7-d252b256b49b_calibrate_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3fcd2e9c-bcb6-4e96-a0a7-d252b256b49b_calibrate_947144_43.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1ec4e725-ff89-4297-bc84-13b85c713e6c_characterizeImage_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/1ec4e725-ff89-4297-bc84-13b85c713e6c_characterizeImage_947144_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcf6dfaf97e1103345f060cdec636877 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/54b4066d-f380-496f-b0e1-61564bb9002d_calibrate_947140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/54b4066d-f380-496f-b0e1-61564bb9002d_calibrate_947140_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da5d3ffe96326a99216c898785e9bfaa with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash a2aa6136b59009133f0bceabd067176f parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/971e205c-994a-49b1-b054-1f2f4ab31b7b_characterizeImage_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/971e205c-994a-49b1-b054-1f2f4ab31b7b_characterizeImage_947246_31.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d80fc55e8a14d8dde70c6bfe2cc38a with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/493681bd-5070-49f5-9b46-5eccb7c5bd67_calibrate_947142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/493681bd-5070-49f5-9b46-5eccb7c5bd67_calibrate_947142_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dea49fbc1f90ce1130c033e394f121cd with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/45def7db-fc4d-4121-a062-233cd9224108_calibrate_947144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/45def7db-fc4d-4121-a062-233cd9224108_calibrate_947144_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b30dceb1f78e6349764c815aaf7f00 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/77cd0ca7-c0cd-4890-b2fe-4293ea9f1d9f_calibrate_947248_21.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/77cd0ca7-c0cd-4890-b2fe-4293ea9f1d9f_calibrate_947248_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:08 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65789c1e6eca2236b1fd23b58590fe10 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 5ef34bde40d9ef8d827fbcfcf90eec60 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d7dfb965-ddc4-426f-89f9-2ddf8bd30222_calibrate_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d7dfb965-ddc4-426f-89f9-2ddf8bd30222_calibrate_947246_28.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f6be0c1f-685b-4bbe-8b03-bcc47acbc8b8_characterizeImage_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/f6be0c1f-685b-4bbe-8b03-bcc47acbc8b8_characterizeImage_947246_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:09 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c01d34e4e08e339766265f9456eaba with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 84c93992276b36b4c40b64552673a5bb parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e6baae1c-3c11-4478-8af0-849576e11ecf_calibrate_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e6baae1c-3c11-4478-8af0-849576e11ecf_calibrate_947144_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e64685b9-1795-463a-ab57-e1c39bb667db_characterizeImage_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/e64685b9-1795-463a-ab57-e1c39bb667db_characterizeImage_947144_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:09 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:09 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 991b4419b35107ac3ae90d80ec10c867 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/e54082fe-c4ec-4e4d-b3a8-cb5cc1d4c455_calibrate_947246_6.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/e54082fe-c4ec-4e4d-b3a8-cb5cc1d4c455_calibrate_947246_6.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2363cfcecdefaf3436b46bbb97c2efa5 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c4ff16c2-7f67-4999-bc39-98e81eb01ad3_calibrate_947246_52.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c4ff16c2-7f67-4999-bc39-98e81eb01ad3_calibrate_947246_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:09 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74fbc4770e003a1e9f9e8fcda53358f with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/65a8c7f2-2ac9-4f2f-b388-f40e094b4261_calibrate_947248_27.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/65a8c7f2-2ac9-4f2f-b388-f40e094b4261_calibrate_947248_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 PD 17031604 PD 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 166 parsl.dataflow.strategy DEBUG: Executor multi has 166 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 166 parsl.dataflow.strategy DEBUG: Executor multi has 166 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97dae25bbac4c0de8472f9c1411926e with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b30d28b0-8837-473e-bd4f-a91976e588e7_calibrate_947244_6.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/b30d28b0-8837-473e-bd4f-a91976e588e7_calibrate_947244_6.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a0db4b208169629651ce0a51c004c9 with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash b8032ddd71d55adff7a877b62a3e27ff parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf687a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/83e46b95-caef-440c-8660-18c1b70a1521_calibrate_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/83e46b95-caef-440c-8660-18c1b70a1521_calibrate_947246_17.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/305313fa-00cf-4969-8cb3-eea84a06f412_characterizeImage_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/305313fa-00cf-4969-8cb3-eea84a06f412_characterizeImage_947246_17.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eca72687cf739bb4e228727d4260597 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9ee2a0d7-04bd-4448-8b0e-16865eabdd8e_calibrate_947248_8.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9ee2a0d7-04bd-4448-8b0e-16865eabdd8e_calibrate_947248_8.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8587dda83241ed5db074085b5172d482 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 8c5b3797d52005a0d1be1792f9ba941c parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f573a4e0-3028-428b-a968-74bbc62e59f4_calibrate_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f573a4e0-3028-428b-a968-74bbc62e59f4_calibrate_947246_18.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0bfd9cf5-61e5-47ca-a9cf-bbc7518da789_characterizeImage_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/0bfd9cf5-61e5-47ca-a9cf-bbc7518da789_characterizeImage_947246_18.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b11247923b284874bdb3be8217107a with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a02d8552-eab5-4b93-87d0-70e99eec3c50_calibrate_947142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a02d8552-eab5-4b93-87d0-70e99eec3c50_calibrate_947142_38.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e04fd45a51719e51ffee84b04cfd70b with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash da947deb30359c22e5bc0dd89d44a848 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/599a3a88-a8c4-4af8-84dc-8c9cae98cb5c_calibrate_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/599a3a88-a8c4-4af8-84dc-8c9cae98cb5c_calibrate_947248_41.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/75691a43-86dc-48ce-81af-a2317213c453_characterizeImage_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/75691a43-86dc-48ce-81af-a2317213c453_characterizeImage_947248_41.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a07adc291938ecee21fc7dee7984d26 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/97a229e8-736b-4831-9be7-5be60fd51d6f_calibrate_947248_49.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/97a229e8-736b-4831-9be7-5be60fd51d6f_calibrate_947248_49.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cd839efb5ccf1cd44bb356b45b5c956 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7fc1a95d-64ed-495c-9142-dfd5af27fd1c_calibrate_947246_55.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f7948506536ea1c0daa1f6b65facb3 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/381cee70-3684-4720-9878-72a3fb19448b_calibrate_947244_38.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/381cee70-3684-4720-9878-72a3fb19448b_calibrate_947244_38.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfba2577e0367ac17b49dab803049c59 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 35dc065acea18ad8d851595706ab9e10 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/998dc5b4-2d68-4bee-85ee-19534ee05944_calibrate_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/998dc5b4-2d68-4bee-85ee-19534ee05944_calibrate_947144_37.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/349e00a5-5675-481c-aeeb-149b69b120d3_characterizeImage_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/349e00a5-5675-481c-aeeb-149b69b120d3_characterizeImage_947144_37.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514cd33e4512b2fe2e7fe1257d58f5bf with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd0e39d1-e90c-4c61-97ff-5f8a9d5a6826_calibrate_947244_1.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd0e39d1-e90c-4c61-97ff-5f8a9d5a6826_calibrate_947244_1.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aae9e8371e85ef20b64fab4a7197c794 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/71740075-a2dc-48ab-94cd-79c5103dcf23_calibrate_947144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/71740075-a2dc-48ab-94cd-79c5103dcf23_calibrate_947144_9.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a375b60dfd0082a42308eb4454ca35b with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 7eb1440517041c2e584d6412263e80b3 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67fa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8746536d-d4fb-4376-8008-477aa5764e64_calibrate_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8746536d-d4fb-4376-8008-477aa5764e64_calibrate_947144_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0bffa731-dda1-4f01-a802-444f86957dc0_characterizeImage_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/0bffa731-dda1-4f01-a802-444f86957dc0_characterizeImage_947144_22.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a51eb4d97dc418407fd6b1f8c7d58ef with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/dfb055f4-3e22-457d-b03a-a18b7f0f07e5_calibrate_947140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/dfb055f4-3e22-457d-b03a-a18b7f0f07e5_calibrate_947140_56.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5bf5a39c1352bbca675fd520e9717d with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08de91c5-6287-4ff5-a599-8cf216f4d307_calibrate_947248_60.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08de91c5-6287-4ff5-a599-8cf216f4d307_calibrate_947248_60.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be7bdcdbfa06c6bd95f5ab961204e990 with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash db1a6598790fa25e6e458e911c46f68d parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/5dbf59d3-f5e9-4138-8a43-2a0f871245dd_calibrate_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/5dbf59d3-f5e9-4138-8a43-2a0f871245dd_calibrate_947248_16.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f61727e2-050e-403a-8e94-de7ca4025c18_characterizeImage_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/f61727e2-050e-403a-8e94-de7ca4025c18_characterizeImage_947248_16.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e8fb9ebce46d4baa9779b3dd51139b3 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8b2ed851-662c-49c1-a78f-4c188699cfcb_calibrate_947246_48.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8b2ed851-662c-49c1-a78f-4c188699cfcb_calibrate_947246_48.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7765451958e22ffbd9ed4848351d1279 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 4a7e4a4e313a2e91e92610bc729bebdd parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e3c8806b-fd3a-4e39-9cbe-adbe4620ceb5_calibrate_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e3c8806b-fd3a-4e39-9cbe-adbe4620ceb5_calibrate_947248_51.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/cea7dcb0-48c8-4196-a0ec-2ee5033b1e6c_characterizeImage_947248_51.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63057d9bf45d67c5a79456e417f6568e with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d6ce8e82-335a-49f3-b653-af3489b25335_calibrate_947144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d6ce8e82-335a-49f3-b653-af3489b25335_calibrate_947144_14.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 091b6cb152609fa8065a83a5dad339d0 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d81daa1e-f3c0-41e1-ab3b-d93ef0e6c6bd_calibrate_947140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/d81daa1e-f3c0-41e1-ab3b-d93ef0e6c6bd_calibrate_947140_16.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28c8fd81c2496ce0c3167f05dde5cfe with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c198debe-7e65-42a2-9a80-ea033ee5f57f_calibrate_947244_10.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c198debe-7e65-42a2-9a80-ea033ee5f57f_calibrate_947244_10.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2591d132eb169153d22efb2f3c5c3267 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 4190e97af9a3466b25f0db07a0002cd8 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/91934b7f-30e4-4e2d-8251-5ae7d5a369ce_calibrate_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/91934b7f-30e4-4e2d-8251-5ae7d5a369ce_calibrate_947144_44.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d14292a0-106d-45fd-a0e3-153d96cca31f_characterizeImage_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/d14292a0-106d-45fd-a0e3-153d96cca31f_characterizeImage_947144_44.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3a8033103bda67eb85bbdd40f622090 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/71df4647-db45-46f7-bf4c-0d31cc3de734_calibrate_947248_18.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/71df4647-db45-46f7-bf4c-0d31cc3de734_calibrate_947248_18.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 151 parsl.dataflow.strategy DEBUG: Executor multi has 151 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2dbd060e2a0ec1616da1272155ec276 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/60dba422-8dcc-4959-9e0f-73507e557e52_calibrate_947244_33.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/60dba422-8dcc-4959-9e0f-73507e557e52_calibrate_947244_33.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 305afc30d587db2bab1b673ce8c66e14 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/19fca630-e4a4-4af5-8c79-49fa018c2e28_calibrate_947248_38.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/19fca630-e4a4-4af5-8c79-49fa018c2e28_calibrate_947248_38.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc33cb20edaef4eddde6b6254bd44a21 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/138563ab-627a-422d-9003-80c417780486_calibrate_947248_23.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/138563ab-627a-422d-9003-80c417780486_calibrate_947248_23.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9885d444eda10e853dac07a7ec076db5 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 1a2e7775810dc82129fa6ade3e144130 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fecf4dbd-7dcc-47d4-b944-aa8912521972_calibrate_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fecf4dbd-7dcc-47d4-b944-aa8912521972_calibrate_947246_54.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a618a2bf-2f6e-4a26-991f-568acf8c13f9_characterizeImage_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/a618a2bf-2f6e-4a26-991f-568acf8c13f9_characterizeImage_947246_54.stderr parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e761bb588380b9eb8016796f66faeb with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/51c97826-623e-4301-98a9-2ed358579b2f_calibrate_947144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/51c97826-623e-4301-98a9-2ed358579b2f_calibrate_947144_48.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3104cb6db8c1a7cf467b196abc5f75c8 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c225ef23-81f9-4d70-8e4d-f0282528e9c0_calibrate_947140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/c225ef23-81f9-4d70-8e4d-f0282528e9c0_calibrate_947140_58.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6723c575ebad82cb12b769697ef6251a with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 21853cf6711c35e699f123b578a00772 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9e0af527-f9dd-485e-a547-5fe312e76627_calibrate_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9e0af527-f9dd-485e-a547-5fe312e76627_calibrate_947142_12.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcd12329-57ce-43f6-ba6b-cea195042a21_characterizeImage_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/bcd12329-57ce-43f6-ba6b-cea195042a21_characterizeImage_947142_12.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45cf573447b6544e7b46831e4ab6d9c with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e14b0cf1-6a3a-4de3-bbd0-1ca30befc1fa_calibrate_947142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/e14b0cf1-6a3a-4de3-bbd0-1ca30befc1fa_calibrate_947142_50.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59fe52c66cbb24a1c2f315cc05d1b1ab with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash cb833fcf1c51ce0337205b5cbdb0465f parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68789a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/480d6999-fabd-4ce2-8acc-c0947ee41c5c_calibrate_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/480d6999-fabd-4ce2-8acc-c0947ee41c5c_calibrate_947244_55.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93cf55d5-6348-422c-8378-04d2a53c0bac_characterizeImage_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/93cf55d5-6348-422c-8378-04d2a53c0bac_characterizeImage_947244_55.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1995428beaf7c255d46d619e3e23421b with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash ac24fafd43953cf511d77c4770a2697a parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1e326de9-cb8b-4e7e-8b4f-625701b401c4_calibrate_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1e326de9-cb8b-4e7e-8b4f-625701b401c4_calibrate_947142_25.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5328ad17-c6a6-4446-a9f0-cac5c57786a7_characterizeImage_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/5328ad17-c6a6-4446-a9f0-cac5c57786a7_characterizeImage_947142_25.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5335be9eb348878bc7cf831886262e84 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 1860f27cf6307ebe9e9d148b6d091d14 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/2480feb5-d7ef-4f8f-8203-55cbd9858def_calibrate_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/2480feb5-d7ef-4f8f-8203-55cbd9858def_calibrate_947244_26.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/dc89e3e0-3787-4bf0-afa7-597964d69f83_characterizeImage_947244_26.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48b491ad56fe715493fad0d9c4274ebf with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 3516b4965983245660c42b91000392cf parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5fd323de-4c31-43f0-99fc-c8869475fb31_calibrate_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5fd323de-4c31-43f0-99fc-c8869475fb31_calibrate_947244_16.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/448bf64b-36f4-404b-8d64-66787d89d7e8_characterizeImage_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/448bf64b-36f4-404b-8d64-66787d89d7e8_characterizeImage_947244_16.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f52793c7bc3a4ed02754b32b3fac740 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9f69d391-d0c1-49a6-8546-6043e274808b_calibrate_947142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9f69d391-d0c1-49a6-8546-6043e274808b_calibrate_947142_52.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8084453cafbb836a2410671b398eddd with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/35adfc82-d3d1-49b7-83b1-a09358e4e621_calibrate_947248_48.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/35adfc82-d3d1-49b7-83b1-a09358e4e621_calibrate_947248_48.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1c493dd0206d7632117106430127a6 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8669e552-1088-4e89-89b0-c1914f2a0425_calibrate_947144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8669e552-1088-4e89-89b0-c1914f2a0425_calibrate_947144_45.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0c7ac1b8f061015978d3a790b83267 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/ed574d67-c97f-41d8-a8ca-bcf14aba7961_calibrate_947244_48.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/ed574d67-c97f-41d8-a8ca-bcf14aba7961_calibrate_947244_48.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2055a17277becb844d3fc4d7137bece with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4a882b26-26c5-4bdc-bb8f-1b39779bf15c_calibrate_947144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4a882b26-26c5-4bdc-bb8f-1b39779bf15c_calibrate_947144_12.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbd748f2978233de1aeaba97390845c with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 021983a27e7c315bd32602de238affb6 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aaccbd3a-6c72-441f-8093-ff28cb0f9c48_calibrate_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aaccbd3a-6c72-441f-8093-ff28cb0f9c48_calibrate_947244_43.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0e5b8386-cd70-4527-8bbc-034dae6bed3b_characterizeImage_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0e5b8386-cd70-4527-8bbc-034dae6bed3b_characterizeImage_947244_43.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b4d440775809e19d4bc6dc873243879 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fd151ad1-5541-4a41-8b47-d53ae59ad987_calibrate_947140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/fd151ad1-5541-4a41-8b47-d53ae59ad987_calibrate_947140_54.stderr parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d24797c3dc1804d4d6f2934badb271 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 6d1cfb8b85ba3e21ebe2a3153f0f5e3b parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/df113a7c-a29c-4c04-bfcd-1b4f118d1922_calibrate_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/df113a7c-a29c-4c04-bfcd-1b4f118d1922_calibrate_947244_15.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1e752b25-dfc1-40bf-bc50-2e574b62b7da_characterizeImage_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/1e752b25-dfc1-40bf-bc50-2e574b62b7da_characterizeImage_947244_15.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 139 parsl.dataflow.strategy DEBUG: Executor multi has 139 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a4699d6e8a68438f6efa13c6ef02dda with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/120286a0-9f22-4a86-8f78-166d39d39582_calibrate_947144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/120286a0-9f22-4a86-8f78-166d39d39582_calibrate_947144_30.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e5f8e08192eb6522425e7f7aff29fb with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0a745fc3-bc8b-4db2-b776-ef1a2c2c634d_calibrate_947248_13.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0a745fc3-bc8b-4db2-b776-ef1a2c2c634d_calibrate_947248_13.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b014200b1d4bd3b09f6b367b1d249975 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash ee505a9aaabb8617afc61fa136c5215f parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/6c9f7ce4-3e28-4caa-888e-fe7c7d3b5ac2_calibrate_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/6c9f7ce4-3e28-4caa-888e-fe7c7d3b5ac2_calibrate_947248_54.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d699e39e-6ab5-4547-bb25-6f6c915b26af_characterizeImage_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/d699e39e-6ab5-4547-bb25-6f6c915b26af_characterizeImage_947248_54.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68999cb9f7baa360a4a500f8f302ed63 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 719a5fb4985d15d09bdc8b481b230292 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/df9bb411-9664-4c1b-8bc9-a2384ff60163_calibrate_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/df9bb411-9664-4c1b-8bc9-a2384ff60163_calibrate_947248_22.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0bab7182-0491-4aff-826f-d1446addde01_characterizeImage_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/0bab7182-0491-4aff-826f-d1446addde01_characterizeImage_947248_22.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c6e7e67c3bd80fd9d002b7037abcc8c with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash b09cc4015904cf8d8e03303e6dae8c01 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d9449c47-93ad-4b55-bc6f-9e8b92f80d57_calibrate_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d9449c47-93ad-4b55-bc6f-9e8b92f80d57_calibrate_947144_17.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/8d1ac713-c61b-4118-ba35-c04340c7a7da_characterizeImage_947144_17.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aa06a468fd9bdb622f1088dbcf5382f with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 65e3f4e8242d46265d67c84bdd4e7237 parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/630c99f8-9618-49f8-ba6c-1c96d654c4ca_calibrate_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/630c99f8-9618-49f8-ba6c-1c96d654c4ca_calibrate_947244_58.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e6ccf20d-fb79-4cae-949f-691ba5c7aeea_characterizeImage_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e6ccf20d-fb79-4cae-949f-691ba5c7aeea_characterizeImage_947244_58.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c601300932a6160afc30ea2411036c99 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash bb65f8169480226e5912a223cc282ab9 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6848900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/239ecbf2-0a99-41bb-b4fa-e853f28d867f_calibrate_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/239ecbf2-0a99-41bb-b4fa-e853f28d867f_calibrate_947140_29.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/826a301c-9300-43c2-a6d7-2964d1928508_characterizeImage_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/826a301c-9300-43c2-a6d7-2964d1928508_characterizeImage_947140_29.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf178cefdede39dcf3880e8f7559c2f with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/56597cb9-7126-4c1e-ae4c-b67f2d4fbc66_calibrate_947144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/56597cb9-7126-4c1e-ae4c-b67f2d4fbc66_calibrate_947144_55.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef0fb4c319f862523c31056f5a67e37c with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fe085fad-5c99-4306-8fdf-196a13cde152_calibrate_947246_51.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fe085fad-5c99-4306-8fdf-196a13cde152_calibrate_947246_51.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c333831c28f50bc6d7b444db3195bdc with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ebe573d1-246d-4352-9be1-57fdc6592ba1_calibrate_947140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/ebe573d1-246d-4352-9be1-57fdc6592ba1_calibrate_947140_35.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 937bee8fc6ca5de8bd773f54d02ea287 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash b730e7f776b1f8053b965eb9b6ef9362 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd2e8ad1-c278-4737-8835-5d5cbb542d32_calibrate_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd2e8ad1-c278-4737-8835-5d5cbb542d32_calibrate_947244_62.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c10ebcf-76d4-44cd-95eb-005a7ea698b5_characterizeImage_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c10ebcf-76d4-44cd-95eb-005a7ea698b5_characterizeImage_947244_62.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f0cc0cc5b51518c8a4b0e9d7eb92af7 with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/cc1471e9-70bd-4643-afa1-82438c4e8234_calibrate_947140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/cc1471e9-70bd-4643-afa1-82438c4e8234_calibrate_947140_9.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61ff28bff6e7e7f30af81575b4ad18c with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/62501877-176a-4017-9528-ecc7dc9737f9_calibrate_947248_14.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/62501877-176a-4017-9528-ecc7dc9737f9_calibrate_947248_14.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3fd0125dc601a95c43f597fa7477ddf with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 918238d2daf781d603bf21367a5df99b parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6634ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/83b357db-7c50-4213-a955-af11575f10f1_calibrate_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/83b357db-7c50-4213-a955-af11575f10f1_calibrate_947140_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/848f30a8-a75d-4479-b047-cae64c51d4e2_characterizeImage_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/848f30a8-a75d-4479-b047-cae64c51d4e2_characterizeImage_947140_47.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2dc5c0f01f74bc9c8110e717ab66e3 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03dc62b8-a844-4b42-a19e-28439403f7d6_calibrate_947248_1.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/03dc62b8-a844-4b42-a19e-28439403f7d6_calibrate_947248_1.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ac44aaeb03db17565df3011959589da with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 9cf9b671776ff2dddde17f2ec3255b66 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/641da7b6-9f76-4ce6-8620-064f783f8e39_calibrate_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/641da7b6-9f76-4ce6-8620-064f783f8e39_calibrate_947246_25.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c2cd7720-34a4-40cc-afcb-3e68b913a7be_characterizeImage_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/c2cd7720-34a4-40cc-afcb-3e68b913a7be_characterizeImage_947246_25.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 131 parsl.dataflow.strategy DEBUG: Executor multi has 131 active tasks, 91/2 running/pending blocks, and 337 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22de05d028962ef158b9fa1954093efa with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash c36a85e797b7944ac303e650aac44de6 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69f7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0833299a-834b-419c-aa45-85d407b8d98c_calibrate_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0833299a-834b-419c-aa45-85d407b8d98c_calibrate_947244_19.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/89d37101-8c3e-46dc-8285-ce7aa7581fba_characterizeImage_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/89d37101-8c3e-46dc-8285-ce7aa7581fba_characterizeImage_947244_19.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c54a11ac66d5869c2858937caf78e0a with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 6b33dc499796b8105c06aeb882e1aa47 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/553b36d7-f7c9-40a2-b440-d352a1e70705_calibrate_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/553b36d7-f7c9-40a2-b440-d352a1e70705_calibrate_947248_42.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/32fd8cfb-c7bb-4898-805d-7ba1cbea1e11_characterizeImage_947248_42.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e406c12c5482dc78124b6d3ebcdfedd7 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 46f55db0098de291d374fd0a3dff7391 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7afb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8483804b-3b38-4929-97fe-d4da0f26ea88_calibrate_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8483804b-3b38-4929-97fe-d4da0f26ea88_calibrate_947244_14.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e74e3349-954d-4241-ab2d-303f4231cfce_characterizeImage_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e74e3349-954d-4241-ab2d-303f4231cfce_characterizeImage_947244_14.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7ea23751c46e057c09d44e2243912e9 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/808b1202-d59c-4dd0-950e-2eb74018723f_calibrate_947244_22.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/808b1202-d59c-4dd0-950e-2eb74018723f_calibrate_947244_22.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7b8a6af679f6a1e6fb1ebf8cb6938eb with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c45ae31e-a06a-4790-a17a-9986a9f66cea_calibrate_947244_29.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c45ae31e-a06a-4790-a17a-9986a9f66cea_calibrate_947244_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 497 try 1 failed parsl.dataflow.dflow ERROR: Task 497 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8c0715e23012d55a55e150a00b51ff02 with result from task 497 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/cf853b0a-ca6b-483d-b6df-5482eeb8bede_calibrate_947248_31.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7043daf8c9aac4ec9e6d4759aa3b83d with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/52d55fdc-a703-445c-86a1-d074f14de7a6_calibrate_947142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/52d55fdc-a703-445c-86a1-d074f14de7a6_calibrate_947142_53.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87168c802d095579cf40dae5556994e9 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b3d64c08-33c9-41db-bc8b-b92ae4c59843_calibrate_947248_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/b3d64c08-33c9-41db-bc8b-b92ae4c59843_calibrate_947248_39.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 085787e6977daeef94283eb2e1bcf09e with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f986bc2a-33be-4d14-b178-16f8f240ca71_calibrate_947248_53.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f986bc2a-33be-4d14-b178-16f8f240ca71_calibrate_947248_53.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c92b8c1bea84ead3c13b170a74de8d53 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/215c7f2c-9548-4227-8e9c-b89df0c70bc5_calibrate_947246_47.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/215c7f2c-9548-4227-8e9c-b89df0c70bc5_calibrate_947246_47.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14eb5b6a1a33ac2b401fe230ef056764 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 6bc51505a44cf3515f71b9bdf44ba8eb parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6920680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3ae7cdfa-f954-49f4-8bb1-bce3b5c9f39e_calibrate_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3ae7cdfa-f954-49f4-8bb1-bce3b5c9f39e_calibrate_947244_44.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/08fb2935-6e8a-46bb-962f-d14bd7388bba_characterizeImage_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/08fb2935-6e8a-46bb-962f-d14bd7388bba_characterizeImage_947244_44.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6f88270e01a9643a76555f5e02dea5f with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7697d1c7-df75-4d69-b3b0-705d37227c87_calibrate_947244_28.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7697d1c7-df75-4d69-b3b0-705d37227c87_calibrate_947244_28.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a24056210ccd856eceba4ca81f572be with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c210eafb-16bc-4c53-80f9-ef7742153656_calibrate_947244_4.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/c210eafb-16bc-4c53-80f9-ef7742153656_calibrate_947244_4.stderr 2024-03-14 05:38:29 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc456ed158a209fedf687087e8c6a7be with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/998a0992-9973-4d1d-9c5c-5c27ef996376_calibrate_947248_46.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/998a0992-9973-4d1d-9c5c-5c27ef996376_calibrate_947248_46.stderr 2024-03-14 05:38:29 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7e3b9daa0606fc0c23b52219169ad30 with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/78d38a3c-3246-4c0a-85ae-5e06f2023c60_calibrate_947244_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/78d38a3c-3246-4c0a-85ae-5e06f2023c60_calibrate_947244_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:30 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:30 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72fced475f992a32372b25a331db3689 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f63d816-7a76-4c5b-84ac-3207b679a88f_calibrate_947248_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f63d816-7a76-4c5b-84ac-3207b679a88f_calibrate_947248_47.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3b3871d08e9aa27df17db18efb01465 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/42b6595c-81ce-421b-954e-8cec390cba3c_calibrate_947246_32.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/42b6595c-81ce-421b-954e-8cec390cba3c_calibrate_947246_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:30 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210303bdf4f0690f692d774846f93d6c with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc9744f2-dd2c-43f6-9bf5-55f2fe3f5269_calibrate_947140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/bc9744f2-dd2c-43f6-9bf5-55f2fe3f5269_calibrate_947140_34.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dea2fbd18d28ce834da06aa67b8fb09 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/056fb217-1440-49b7-a83e-5e942f2bd4ae_calibrate_947246_33.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/056fb217-1440-49b7-a83e-5e942f2bd4ae_calibrate_947246_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:31 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e221758c15b6cb259a9e39d2fa6b5125 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6510f008-6292-4a92-b0c0-7eabed6ab407_calibrate_947246_5.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6510f008-6292-4a92-b0c0-7eabed6ab407_calibrate_947246_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:31 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:32 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a10f4ed8853f3a5e0770413eb4b4169b with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/851eddc7-0153-4409-b91f-78fb52c23c14_calibrate_947244_59.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/851eddc7-0153-4409-b91f-78fb52c23c14_calibrate_947244_59.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894a3ccdadcb7f19dfe96878fc538745 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash c6a31ae8a182439e11c0e4d78d390477 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ac460c04-aace-4026-b889-4db75566f317_calibrate_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ac460c04-aace-4026-b889-4db75566f317_calibrate_947246_57.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/87294cf0-5207-4d53-95bb-98f7c9492212_characterizeImage_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/87294cf0-5207-4d53-95bb-98f7c9492212_characterizeImage_947246_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:32 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 947 try 0 failed parsl.dataflow.dflow INFO: Task 947 marked for retry parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash a2aa6136b59009133f0bceabd067176f parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 1 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:32 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc737d001599680be95cbfec539b999 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash dc7fa5153ddbfb348b55e05ae5e113ea parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9b173b8f-d9f4-4069-8056-203dc62ca2ac_calibrate_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9b173b8f-d9f4-4069-8056-203dc62ca2ac_calibrate_947142_18.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/eca00987-368d-4905-914d-6303ed269e66_characterizeImage_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/eca00987-368d-4905-914d-6303ed269e66_characterizeImage_947142_18.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b8fb32cb84d0fab56fa62ea7cd0bc39 with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cf585163-4ec2-43bb-ae17-f40d358bc177_calibrate_947142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/cf585163-4ec2-43bb-ae17-f40d358bc177_calibrate_947142_16.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb577c64e1176b3e4bc3303e44574425 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 0b600e9066ac86fddcfdf305a54f853b parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/29059608-1770-495b-bad3-fd0c4a033d20_calibrate_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/29059608-1770-495b-bad3-fd0c4a033d20_calibrate_947248_33.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9942bff8-b065-47cd-87a8-3bae65a92a16_characterizeImage_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/9942bff8-b065-47cd-87a8-3bae65a92a16_characterizeImage_947248_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:33 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce17e5c5918614d645caecd817d2fbc5 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash d5f260e262fc184a47b31a154d300ae9 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/14308b75-995f-4a47-8543-8ac23834bb97_calibrate_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/14308b75-995f-4a47-8543-8ac23834bb97_calibrate_947244_12.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/21f73a23-6a44-4c3c-a22b-cb41142d32a7_characterizeImage_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/21f73a23-6a44-4c3c-a22b-cb41142d32a7_characterizeImage_947244_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:33 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:34 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa310b794145c46c9b5f643a81e434e4 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f337863-2ec1-4a82-bba6-9decdb4ec23a_calibrate_947248_7.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/9f337863-2ec1-4a82-bba6-9decdb4ec23a_calibrate_947248_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:34 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:34 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2117f2c63494c448dc0872cd2682d1 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41a80556-6597-41b8-8925-744eea1fb15f_calibrate_947144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/41a80556-6597-41b8-8925-744eea1fb15f_calibrate_947144_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:35 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52907772eef2d5675c29049bedbe7dc3 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f6358306-d4c9-4a58-b6f0-fee41aa1122b_calibrate_947248_57.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f6358306-d4c9-4a58-b6f0-fee41aa1122b_calibrate_947248_57.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 950f10bffef14d45ae27cf2ba92b3bf9 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 4f18db879e08b37c15d11238ef63f400 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6922980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0cf91172-e71e-49df-aee5-16447b51b7d0_calibrate_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0cf91172-e71e-49df-aee5-16447b51b7d0_calibrate_947248_28.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/35a4c333-2047-4bb3-be3e-b8c79f959147_characterizeImage_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/35a4c333-2047-4bb3-be3e-b8c79f959147_characterizeImage_947248_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:35 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7debec2f2541d87d5819d75f66444e52 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash faced31ad749af8fd52e9ad0116c4cb0 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6879da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/e5bd8b45-8338-46d2-94dd-8f2ef2b2e33b_calibrate_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/e5bd8b45-8338-46d2-94dd-8f2ef2b2e33b_calibrate_947244_36.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/291f38a6-e1b7-42fc-b200-44d91cd16ac8_characterizeImage_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/291f38a6-e1b7-42fc-b200-44d91cd16ac8_characterizeImage_947244_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:36 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:36 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c2c3d6a61609e39dbb55b6f1d74aad3 with result from task 1037 parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:36 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3d111534-eb0a-4eee-b91f-5de26cfec652_calibrate_947246_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/3d111534-eb0a-4eee-b91f-5de26cfec652_calibrate_947246_1.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db2635b51e834ff727a788b024660aec with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/41eb3935-8e10-45ea-a87a-aa96c8579991_calibrate_947248_36.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/41eb3935-8e10-45ea-a87a-aa96c8579991_calibrate_947248_36.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0bcd37c85ad1db307e7b7f887657aba with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5c4751a7-0930-4aa8-85da-f0baee89f5d1_calibrate_947142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/5c4751a7-0930-4aa8-85da-f0baee89f5d1_calibrate_947142_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:37 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry accc7a665bb97ab66c69fe3f77c3bf6f with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/740154ec-1a1a-4faf-bdd5-5cddeb58ad53_calibrate_947144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/740154ec-1a1a-4faf-bdd5-5cddeb58ad53_calibrate_947144_56.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ce29abe843601cb9e9483f94329f28d with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/dea719e7-7eff-424a-afb1-3a70a6305d39_calibrate_947244_30.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/dea719e7-7eff-424a-afb1-3a70a6305d39_calibrate_947244_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:37 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef644e9f2394c41444852464bbda0389 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 6f9f52a96a9c62df71f9ae9f22d33c25 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68796c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ac6ad650-6327-47de-80a7-0654fe27e581_calibrate_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ac6ad650-6327-47de-80a7-0654fe27e581_calibrate_947248_29.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8bf9b82f-0292-4e4e-8bd7-6b0495794cce_characterizeImage_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8bf9b82f-0292-4e4e-8bd7-6b0495794cce_characterizeImage_947248_29.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5036a03ceaa4e4e6ed530d67e4439640 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6287d4e1-46c5-472d-9b88-7a4c475f5d84_calibrate_947246_35.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6287d4e1-46c5-472d-9b88-7a4c475f5d84_calibrate_947246_35.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d77a7567b1df7cf4a89961859f0bd0d8 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a7d14516-90db-4aac-80c6-4c5161ed4b87_calibrate_947144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/a7d14516-90db-4aac-80c6-4c5161ed4b87_calibrate_947144_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:38 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3a7c1a05b2f9aafb18c82ff306226c with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 451d27fc72125a0fb603589aba416226 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b85f4ce1-e305-4764-bf9f-b5d65d438a0a_calibrate_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b85f4ce1-e305-4764-bf9f-b5d65d438a0a_calibrate_947246_12.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/ba7c84e7-6ddf-4f90-bfd1-97db2ea18307_characterizeImage_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/ba7c84e7-6ddf-4f90-bfd1-97db2ea18307_characterizeImage_947246_12.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c28a89b5f71df0c1bcd677f13c364513 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5aa85f43-4d29-43ac-8e7e-5e81de6e71a7_calibrate_947244_3.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5aa85f43-4d29-43ac-8e7e-5e81de6e71a7_calibrate_947244_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:38 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b90552d3d9492ed2f853cac9db3737 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ca047a3c-890d-470d-9ec5-7f0435d152a7_calibrate_947248_34.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ca047a3c-890d-470d-9ec5-7f0435d152a7_calibrate_947248_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:38 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17cc871b8f06384a586a46d95e41a20d with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 70de9b5037fb7b3b4311fa61798ad96c parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf671f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/53857d8a-8763-4b5e-8bfd-db5a0540de60_calibrate_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/53857d8a-8763-4b5e-8bfd-db5a0540de60_calibrate_947246_42.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b16784f5-f48c-42f1-a251-f76f1b3b0651_characterizeImage_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/b16784f5-f48c-42f1-a251-f76f1b3b0651_characterizeImage_947246_42.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c47bd21e6306e87b475dba117c338e17 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash ac13ea1a0d3aced85fd0d2c074d085db parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68d1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2d25828d-3597-4038-a012-831458119ffd_calibrate_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2d25828d-3597-4038-a012-831458119ffd_calibrate_947248_20.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/69a7f9be-e877-4b96-8ffc-efd67708d9db_characterizeImage_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/69a7f9be-e877-4b96-8ffc-efd67708d9db_characterizeImage_947248_20.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67ec77225c31fb32ea654d046a591ba1 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash db08b5fa9730233171377e68f2b584be parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ad44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/eb820854-2cfb-429e-9adb-73a33dc23cef_calibrate_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/eb820854-2cfb-429e-9adb-73a33dc23cef_calibrate_947248_12.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8be9c07f-f44a-48a4-a937-9ed5ffaedee0_characterizeImage_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/8be9c07f-f44a-48a4-a937-9ed5ffaedee0_characterizeImage_947248_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:39 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57e6c9ef0709d4226c5102872fe7b1d2 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 808f390f840126c6e36c37c2715b29ee parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c587049-ce14-447b-929c-a344b8e178fa_characterizeImage_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/0c587049-ce14-447b-929c-a344b8e178fa_characterizeImage_947244_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:39 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed7ff1f665391a73598479e7af4d083b with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c18dd84d-34c7-4cdd-a904-aeac876dd66c_calibrate_947246_53.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c18dd84d-34c7-4cdd-a904-aeac876dd66c_calibrate_947246_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:39 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:40 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:40 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8532e1f5fb24d2db9d2ce53a30dd546 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/6e8cf3a8-95a9-4a02-8111-845617457cdd_calibrate_947244_49.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/6e8cf3a8-95a9-4a02-8111-845617457cdd_calibrate_947244_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:41 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b284d56e5949d9b9f8a924f342cb327f with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7033ce03-0969-4f6e-be1d-f576cb6a975c_calibrate_947246_56.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/7033ce03-0969-4f6e-be1d-f576cb6a975c_calibrate_947246_56.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa60c8a845a3677312bafe69b09e9804 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/629a6c57-c4b6-43cf-b0bb-56e3a63d8dc7_calibrate_947144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/629a6c57-c4b6-43cf-b0bb-56e3a63d8dc7_calibrate_947144_42.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f01b3c9cb099481d5787e68f84ea7d42 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/37eba8b1-2050-4e1d-84b3-69565b130a3b_calibrate_947142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/37eba8b1-2050-4e1d-84b3-69565b130a3b_calibrate_947142_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:41 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2a1928eb6711811f9bceb8a73b8b819 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3fcd2e9c-bcb6-4e96-a0a7-d252b256b49b_calibrate_947144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/3fcd2e9c-bcb6-4e96-a0a7-d252b256b49b_calibrate_947144_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:41 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:42 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:42 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b45b1e9e785c7a2cdec95912a0dd545 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3933210d-554d-4fc7-9fdf-12639f9d59ec_calibrate_947244_27.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3933210d-554d-4fc7-9fdf-12639f9d59ec_calibrate_947244_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:42 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:43 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154b5ecb7ef8d47be756b148e0e08968 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/91bc6497-f2dd-4df0-a4a5-1585c804e21a_calibrate_947248_19.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/91bc6497-f2dd-4df0-a4a5-1585c804e21a_calibrate_947248_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:43 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c50b491b72369c34d46143ff7bb56daa with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1fea23fa-17e7-4d66-aeb6-b16aef280524_calibrate_947142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1fea23fa-17e7-4d66-aeb6-b16aef280524_calibrate_947142_44.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83a06a13241fdf19ecf8945b20bf6f7a with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/986aeea4-f053-4e83-ad18-133fda3d12a6_calibrate_947244_13.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/986aeea4-f053-4e83-ad18-133fda3d12a6_calibrate_947244_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:43 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69de4b4dfb4ac0a4a177a625168dcbd6 with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1a851e7d-7ccc-4df5-ba71-b42ad7502473_calibrate_947142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1a851e7d-7ccc-4df5-ba71-b42ad7502473_calibrate_947142_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:44 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13d0aa91ce7d867e1c1a7402d087bf2e with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 4e81f222df530ad906263613e39bc07d parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf684b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/37b6b1bd-b64a-4fe5-b373-824875a7cbc2_calibrate_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/37b6b1bd-b64a-4fe5-b373-824875a7cbc2_calibrate_947246_29.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/54f06075-a88b-483a-b646-c2b3aabe2115_characterizeImage_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/54f06075-a88b-483a-b646-c2b3aabe2115_characterizeImage_947246_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:44 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e967e61869e718fb3be5f2000a0236f with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e355a4c1-f590-4915-8cfe-773cdcb2faca_calibrate_947248_32.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e355a4c1-f590-4915-8cfe-773cdcb2faca_calibrate_947248_32.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95466560bcf9dcdbc1df3079f8a2e7ff with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6276ca36-ca07-4252-97ee-c5d7ffcc80ad_calibrate_947246_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/6276ca36-ca07-4252-97ee-c5d7ffcc80ad_calibrate_947246_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:45 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:45 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:45 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:46 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1aa76fc976e759bb6925a40adeaac7e with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 276b51a5122e8985d14a457dc9512583 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7a940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/ecbbbc65-9479-4ca9-bb02-eaa947980b60_calibrate_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/ecbbbc65-9479-4ca9-bb02-eaa947980b60_calibrate_947142_7.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/95326bfc-1440-44d0-9f33-9e6d0d1dd9c9_characterizeImage_947142_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:46 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac24fafd43953cf511d77c4770a2697a with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1e326de9-cb8b-4e7e-8b4f-625701b401c4_calibrate_947142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/1e326de9-cb8b-4e7e-8b4f-625701b401c4_calibrate_947142_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:46 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa7306644f7ffd6aaa38610da94aaf7 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bc834afc-cc7d-4487-b811-4128e1a91c44_calibrate_947248_24.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/bc834afc-cc7d-4487-b811-4128e1a91c44_calibrate_947248_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:47 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00dd44d82df430ed76cb67ea5d0bd7a2 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d336f693-7add-4be1-9b05-7db32542213d_calibrate_947244_50.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d336f693-7add-4be1-9b05-7db32542213d_calibrate_947244_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:47 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:47 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d051109662b8b8896ea669acdb27f818 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/51d3d5fb-81e5-4385-8a2f-2c1562142007_calibrate_947246_4.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/51d3d5fb-81e5-4385-8a2f-2c1562142007_calibrate_947246_4.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df3b341f438b9d2232f2d4d26496aec3 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/994d60c9-4b22-4b80-a47b-78f823f93368_calibrate_947244_54.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/994d60c9-4b22-4b80-a47b-78f823f93368_calibrate_947244_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:48 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb1440517041c2e584d6412263e80b3 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8746536d-d4fb-4376-8008-477aa5764e64_calibrate_947144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/8746536d-d4fb-4376-8008-477aa5764e64_calibrate_947144_22.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7cf50c8d30902d12367dbb09a7cabe2 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8c56c037-feeb-47d9-ab8a-eac095c7b8e5_calibrate_947246_50.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/8c56c037-feeb-47d9-ab8a-eac095c7b8e5_calibrate_947246_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:48 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb65f8169480226e5912a223cc282ab9 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/239ecbf2-0a99-41bb-b4fa-e853f28d867f_calibrate_947140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/239ecbf2-0a99-41bb-b4fa-e853f28d867f_calibrate_947140_29.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4190e97af9a3466b25f0db07a0002cd8 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/91934b7f-30e4-4e2d-8251-5ae7d5a369ce_calibrate_947144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/91934b7f-30e4-4e2d-8251-5ae7d5a369ce_calibrate_947144_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:49 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:49 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:49 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:50 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7cf127fbbc6f6dbd46369359e168faa with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f9cb3895-9919-45ca-9484-1f765d8de649_calibrate_947142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/f9cb3895-9919-45ca-9484-1f765d8de649_calibrate_947142_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:50 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03f0e96ba119876434d77686460798c4 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/591555c0-26d0-4671-82d4-30d22bffd6b2_calibrate_947244_60.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/591555c0-26d0-4671-82d4-30d22bffd6b2_calibrate_947244_60.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d095e09d00e3239f791598dc3379a5 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 52068806b39f846eaf7cf1e0baf16da4 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6921620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 1123 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/48c14552-2799-4747-be23-ead5984d9ff6_calibrate_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/48c14552-2799-4747-be23-ead5984d9ff6_calibrate_947244_25.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e21cdfe1-4e0d-4ffd-8160-4641094fd3af_characterizeImage_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/e21cdfe1-4e0d-4ffd-8160-4641094fd3af_characterizeImage_947244_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:50 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1caef8a9af0410d6ec7b9f4564a29753 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/61eef436-f546-4785-8417-3d7ef1cc8c62_calibrate_947246_23.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/61eef436-f546-4785-8417-3d7ef1cc8c62_calibrate_947246_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:51 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:51 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1073b1ce65ec97af99a4fab9a3b2199 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/cbdf93b2-1e44-4b9f-b2e0-b523fd8e3a70_calibrate_947244_17.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/cbdf93b2-1e44-4b9f-b2e0-b523fd8e3a70_calibrate_947244_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:52 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2b6cc7325078d46ec0e2de60a7f251 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/89447251-635b-443d-bf21-31ef2ab88589_calibrate_947248_50.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/89447251-635b-443d-bf21-31ef2ab88589_calibrate_947248_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:52 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:52 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dcd709f495cdfc1e03816b9ac9a19d3 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/38acd200-0320-4aab-a4f8-75e3c820c6bd_calibrate_947246_20.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/38acd200-0320-4aab-a4f8-75e3c820c6bd_calibrate_947246_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:53 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:53 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d562f55345650655814e6eedd50ccd9b with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/85bd69cf-06ed-4917-a30e-859a67e3ba3e_calibrate_947246_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/85bd69cf-06ed-4917-a30e-859a67e3ba3e_calibrate_947246_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:53 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:54 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35dc065acea18ad8d851595706ab9e10 with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/998dc5b4-2d68-4bee-85ee-19534ee05944_calibrate_947144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/998dc5b4-2d68-4bee-85ee-19534ee05944_calibrate_947144_37.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 563fad703e0f85e9f902cb4941909d47 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/72b43d55-4d0e-4492-998f-9cf01ede155c_calibrate_947244_42.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/72b43d55-4d0e-4492-998f-9cf01ede155c_calibrate_947244_42.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7682ee9e6f9baf9febbe94911d06e21 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4dec69b5-4585-4b1e-aea2-cbc6c14f4b82_calibrate_947244_46.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4dec69b5-4585-4b1e-aea2-cbc6c14f4b82_calibrate_947244_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:54 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21853cf6711c35e699f123b578a00772 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9e0af527-f9dd-485e-a547-5fe312e76627_calibrate_947142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9e0af527-f9dd-485e-a547-5fe312e76627_calibrate_947142_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:54 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b363b238c787787ac7bd2410a156ff46 with result from task 1007 parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:55 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/de0caadc-ade0-4ffe-b44f-5d4b4c247e75_calibrate_947244_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/de0caadc-ade0-4ffe-b44f-5d4b4c247e75_calibrate_947244_40.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b67677967d369a2c9b8a006ab456303b with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8f2b7306-9527-4a3f-9259-459ca4ea7ba7_calibrate_947244_32.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8f2b7306-9527-4a3f-9259-459ca4ea7ba7_calibrate_947244_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:55 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e27fcb251f366dc7284ed7139a065016 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7e7f8eaf-bfda-4ffb-9a4c-bcc68bfe1b64_calibrate_947248_35.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/7e7f8eaf-bfda-4ffb-9a4c-bcc68bfe1b64_calibrate_947248_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:56 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918238d2daf781d603bf21367a5df99b with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/83b357db-7c50-4213-a955-af11575f10f1_calibrate_947140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/83b357db-7c50-4213-a955-af11575f10f1_calibrate_947140_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:56 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:56 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:57 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:57 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:57 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81d0c67e33972fe2bec470382f84ff9f with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 8be81783b83f5bd200434d5747484136 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 1124 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c33a4f2b-5688-4d1f-9d67-5cd9b1887c67_calibrate_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c33a4f2b-5688-4d1f-9d67-5cd9b1887c67_calibrate_947248_10.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/26ea386f-3e9c-4db1-ab61-34e4a65308b1_characterizeImage_947248_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:58 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9330b63ad1feaa1af764141106898e12 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c748bf09-0994-42ba-9ee5-5a58b6a614ef_calibrate_947246_59.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/c748bf09-0994-42ba-9ee5-5a58b6a614ef_calibrate_947246_59.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c93992276b36b4c40b64552673a5bb with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e6baae1c-3c11-4478-8af0-849576e11ecf_calibrate_947144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/e6baae1c-3c11-4478-8af0-849576e11ecf_calibrate_947144_15.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e025ed1bffea5938983458db9634c68 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 5348ff98fd7c6cd1ffb1abe7bdc83912 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf66362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1125 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f011e314-f0ef-4d37-9ed4-cf0f704bd72c_calibrate_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f011e314-f0ef-4d37-9ed4-cf0f704bd72c_calibrate_947246_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/99af14a2-97b8-4c00-9e2f-f62b17323b4e_characterizeImage_947246_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:58 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:58 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a7e4a4e313a2e91e92610bc729bebdd with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e3c8806b-fd3a-4e39-9cbe-adbe4620ceb5_calibrate_947248_51.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/e3c8806b-fd3a-4e39-9cbe-adbe4620ceb5_calibrate_947248_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:59 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa16f2a8b3457a6c0c6cf2ac4d403ce with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7a3d6cd7-aed0-4f97-b412-8cba5196799c_calibrate_947244_11.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/7a3d6cd7-aed0-4f97-b412-8cba5196799c_calibrate_947244_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:38:59 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e35731b3a3c53c77045ed1a7d0f407ce with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a021663c-9362-4cf5-bc8c-e9172d867052_calibrate_947246_38.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/a021663c-9362-4cf5-bc8c-e9172d867052_calibrate_947246_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:00 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 947 try 1 failed parsl.dataflow.dflow ERROR: Task 947 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a2aa6136b59009133f0bceabd067176f with result from task 947 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/babce4f4-3b1e-406b-88cc-2526fbfcd81a_calibrate_947246_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:00 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beaf6488041fc13c1d52b7f0730e95d5 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/4ef956d3-365f-4209-9df3-f02e0e8545b0_calibrate_947248_62.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/4ef956d3-365f-4209-9df3-f02e0e8545b0_calibrate_947248_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:01 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:01 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:01 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:02 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 131e0570f61425a23a7b90a6fdde42ef with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c035ae5f-5590-41b0-b2ab-dfe0b0728d7a_calibrate_947248_6.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c035ae5f-5590-41b0-b2ab-dfe0b0728d7a_calibrate_947248_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:02 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b09cc4015904cf8d8e03303e6dae8c01 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d9449c47-93ad-4b55-bc6f-9e8b92f80d57_calibrate_947144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/d9449c47-93ad-4b55-bc6f-9e8b92f80d57_calibrate_947144_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:02 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf61c570d91143b17e1d4bec6de11fae with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/829f4265-b8e5-4264-b97c-a7e02f35615a_calibrate_947246_24.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/829f4265-b8e5-4264-b97c-a7e02f35615a_calibrate_947246_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:03 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:03 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:04 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8032ddd71d55adff7a877b62a3e27ff with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/83e46b95-caef-440c-8660-18c1b70a1521_calibrate_947246_17.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/83e46b95-caef-440c-8660-18c1b70a1521_calibrate_947246_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 53 parsl.dataflow.strategy DEBUG: Executor multi has 53 active tasks, 93/0 running/pending blocks, and 345 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:39:04 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc7fa5153ddbfb348b55e05ae5e113ea with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9b173b8f-d9f4-4069-8056-203dc62ca2ac_calibrate_947142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/9b173b8f-d9f4-4069-8056-203dc62ca2ac_calibrate_947142_18.stderr 2024-03-14 05:39:04 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d30ed791c3878755007cc251ba79f7b4 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f5efcaa6-592a-468f-a02a-4c9b28fc5882_calibrate_947248_52.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/f5efcaa6-592a-468f-a02a-4c9b28fc5882_calibrate_947248_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:05 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c5b3797d52005a0d1be1792f9ba941c with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f573a4e0-3028-428b-a968-74bbc62e59f4_calibrate_947246_18.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f573a4e0-3028-428b-a968-74bbc62e59f4_calibrate_947246_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:05 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1c12ed045b14e9de9a86e9613ea0a6 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4a865334-ca9b-4cdb-8020-b9537c1d4dc3_calibrate_947244_7.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/4a865334-ca9b-4cdb-8020-b9537c1d4dc3_calibrate_947244_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:05 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:06 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 705405b02f271d623a6c58b53433d5e3 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash c115aa6b4e080a648d57e5a4be2ed663 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67f8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1126 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4e7ae4a3-b641-491e-8972-c98dd0b2452f_calibrate_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4e7ae4a3-b641-491e-8972-c98dd0b2452f_calibrate_947246_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7df68f6b-2f1b-490b-a775-9408ddbb475f_characterizeImage_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/7df68f6b-2f1b-490b-a775-9408ddbb475f_characterizeImage_947246_37.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84eb2e3e74fd0e9e3e40840fc035ddc8 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/890278a1-1a93-4f47-bedf-3d3c69645429_calibrate_947244_41.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/890278a1-1a93-4f47-bedf-3d3c69645429_calibrate_947244_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:06 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:06 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:07 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:07 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:07 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:07 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f012394d71a966f59d3deadc23b5614 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 5d81147e54d0fcd3d3ecf259898db2a6 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69237e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 1127 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/3bffb537-783e-448d-954d-b8bf40bf2920_calibrate_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/3bffb537-783e-448d-954d-b8bf40bf2920_calibrate_947248_25.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/50fe245e-f669-433c-91ed-87bd304916d6_characterizeImage_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947248/50fe245e-f669-433c-91ed-87bd304916d6_characterizeImage_947248_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:08 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:08 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:09 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:09 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:09 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb833fcf1c51ce0337205b5cbdb0465f with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/480d6999-fabd-4ce2-8acc-c0947ee41c5c_calibrate_947244_55.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/480d6999-fabd-4ce2-8acc-c0947ee41c5c_calibrate_947244_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:09 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7af165a119e62411e9e37887de62e45a with result from task 830 parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:09 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ad3e5756-252c-44d5-9672-073b3c8f4375_calibrate_947248_9.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ad3e5756-252c-44d5-9672-073b3c8f4375_calibrate_947248_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 671 try 0 failed parsl.dataflow.dflow INFO: Task 671 marked for retry parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 808f390f840126c6e36c37c2715b29ee parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6849260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 1 launched on executor multi with executor id 1128 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:10 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:11 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dc51a86956333cd2f9193e0ef0d8af8 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08ad0592-6147-4c40-a192-fded2e06f0a4_calibrate_947248_11.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/08ad0592-6147-4c40-a192-fded2e06f0a4_calibrate_947248_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12408b6f50a0032f0ad690d393aaadba with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bbe367db-b29a-4ac8-8b4e-9597bd7a03a0_calibrate_947244_20.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bbe367db-b29a-4ac8-8b4e-9597bd7a03a0_calibrate_947244_20.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2e7775810dc82129fa6ade3e144130 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fecf4dbd-7dcc-47d4-b944-aa8912521972_calibrate_947246_54.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/fecf4dbd-7dcc-47d4-b944-aa8912521972_calibrate_947246_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:12 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c36a85e797b7944ac303e650aac44de6 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0833299a-834b-419c-aa45-85d407b8d98c_calibrate_947244_19.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/0833299a-834b-419c-aa45-85d407b8d98c_calibrate_947244_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:13 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef34bde40d9ef8d827fbcfcf90eec60 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d7dfb965-ddc4-426f-89f9-2ddf8bd30222_calibrate_947246_28.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/d7dfb965-ddc4-426f-89f9-2ddf8bd30222_calibrate_947246_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3516b4965983245660c42b91000392cf with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5fd323de-4c31-43f0-99fc-c8869475fb31_calibrate_947244_16.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/5fd323de-4c31-43f0-99fc-c8869475fb31_calibrate_947244_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:14 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac41395f7566a7b505002cdccbcec670 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d95c46c9-a77b-4420-bcbf-d3c499041213_calibrate_947244_18.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/d95c46c9-a77b-4420-bcbf-d3c499041213_calibrate_947244_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da947deb30359c22e5bc0dd89d44a848 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/599a3a88-a8c4-4af8-84dc-8c9cae98cb5c_calibrate_947248_41.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/599a3a88-a8c4-4af8-84dc-8c9cae98cb5c_calibrate_947248_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:15 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d1cfb8b85ba3e21ebe2a3153f0f5e3b with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/df113a7c-a29c-4c04-bfcd-1b4f118d1922_calibrate_947244_15.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/df113a7c-a29c-4c04-bfcd-1b4f118d1922_calibrate_947244_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:16 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db487e2be836737d82bf154221c4d7ba with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c166304b-0652-4d4f-9772-419b0f09e2e9_calibrate_947248_61.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c166304b-0652-4d4f-9772-419b0f09e2e9_calibrate_947248_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:17 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:18 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:18 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:18 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:18 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 36 parsl.dataflow.strategy DEBUG: Executor multi has 36 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 36 parsl.dataflow.strategy DEBUG: Executor multi has 36 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db1a6598790fa25e6e458e911c46f68d with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/5dbf59d3-f5e9-4138-8a43-2a0f871245dd_calibrate_947248_16.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/5dbf59d3-f5e9-4138-8a43-2a0f871245dd_calibrate_947248_16.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b730e7f776b1f8053b965eb9b6ef9362 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd2e8ad1-c278-4737-8835-5d5cbb542d32_calibrate_947244_62.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/bd2e8ad1-c278-4737-8835-5d5cbb542d32_calibrate_947244_62.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276b51a5122e8985d14a457dc9512583 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/ecbbbc65-9479-4ca9-bb02-eaa947980b60_calibrate_947142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/ecbbbc65-9479-4ca9-bb02-eaa947980b60_calibrate_947142_7.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65e3f4e8242d46265d67c84bdd4e7237 with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/630c99f8-9618-49f8-ba6c-1c96d654c4ca_calibrate_947244_58.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/630c99f8-9618-49f8-ba6c-1c96d654c4ca_calibrate_947244_58.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46f55db0098de291d374fd0a3dff7391 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8483804b-3b38-4929-97fe-d4da0f26ea88_calibrate_947244_14.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/8483804b-3b38-4929-97fe-d4da0f26ea88_calibrate_947244_14.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 021983a27e7c315bd32602de238affb6 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aaccbd3a-6c72-441f-8093-ff28cb0f9c48_calibrate_947244_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aaccbd3a-6c72-441f-8093-ff28cb0f9c48_calibrate_947244_43.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 30 parsl.dataflow.strategy DEBUG: Executor multi has 30 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee505a9aaabb8617afc61fa136c5215f with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/6c9f7ce4-3e28-4caa-888e-fe7c7d3b5ac2_calibrate_947248_54.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/6c9f7ce4-3e28-4caa-888e-fe7c7d3b5ac2_calibrate_947248_54.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc51505a44cf3515f71b9bdf44ba8eb with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3ae7cdfa-f954-49f4-8bb1-bce3b5c9f39e_calibrate_947244_44.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/3ae7cdfa-f954-49f4-8bb1-bce3b5c9f39e_calibrate_947244_44.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b33dc499796b8105c06aeb882e1aa47 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/553b36d7-f7c9-40a2-b440-d352a1e70705_calibrate_947248_42.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/553b36d7-f7c9-40a2-b440-d352a1e70705_calibrate_947248_42.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f18db879e08b37c15d11238ef63f400 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0cf91172-e71e-49df-aee5-16447b51b7d0_calibrate_947248_28.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/0cf91172-e71e-49df-aee5-16447b51b7d0_calibrate_947248_28.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cf9b671776ff2dddde17f2ec3255b66 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/641da7b6-9f76-4ce6-8620-064f783f8e39_calibrate_947246_25.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/641da7b6-9f76-4ce6-8620-064f783f8e39_calibrate_947246_25.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1860f27cf6307ebe9e9d148b6d091d14 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/2480feb5-d7ef-4f8f-8203-55cbd9858def_calibrate_947244_26.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/2480feb5-d7ef-4f8f-8203-55cbd9858def_calibrate_947244_26.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70de9b5037fb7b3b4311fa61798ad96c with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/53857d8a-8763-4b5e-8bfd-db5a0540de60_calibrate_947246_42.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/53857d8a-8763-4b5e-8bfd-db5a0540de60_calibrate_947246_42.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc2562e285403bf8e668f290b7cda223 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash e5513129da84951d97039c76833a1e1e parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf67acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 1129 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b043a2e1-064a-46a1-a67d-3e4d8aaff544_calibrate_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b043a2e1-064a-46a1-a67d-3e4d8aaff544_calibrate_947246_61.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d16653f3-488d-41e2-b62c-25067e2e57db_characterizeImage_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947246/d16653f3-488d-41e2-b62c-25067e2e57db_characterizeImage_947246_61.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faced31ad749af8fd52e9ad0116c4cb0 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/e5bd8b45-8338-46d2-94dd-8f2ef2b2e33b_calibrate_947244_36.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/e5bd8b45-8338-46d2-94dd-8f2ef2b2e33b_calibrate_947244_36.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 634264f3b90eac6b0eb6f26d1448d053 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 69211dd6cc4002a9f2cef7c34fea5f5b parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1130 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d78b98db-0edc-42c1-b1f8-73f3de4358ec_calibrate_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d78b98db-0edc-42c1-b1f8-73f3de4358ec_calibrate_947142_4.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2385e5a6-f900-4967-bf6e-e34665695709_characterizeImage_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/2385e5a6-f900-4967-bf6e-e34665695709_characterizeImage_947142_4.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b600e9066ac86fddcfdf305a54f853b with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/29059608-1770-495b-bad3-fd0c4a033d20_calibrate_947248_33.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/29059608-1770-495b-bad3-fd0c4a033d20_calibrate_947248_33.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 21 parsl.dataflow.strategy DEBUG: Executor multi has 21 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 719a5fb4985d15d09bdc8b481b230292 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/df9bb411-9664-4c1b-8bc9-a2384ff60163_calibrate_947248_22.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/df9bb411-9664-4c1b-8bc9-a2384ff60163_calibrate_947248_22.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db08b5fa9730233171377e68f2b584be with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/eb820854-2cfb-429e-9adb-73a33dc23cef_calibrate_947248_12.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/eb820854-2cfb-429e-9adb-73a33dc23cef_calibrate_947248_12.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 671 try 1 failed parsl.dataflow.dflow ERROR: Task 671 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 808f390f840126c6e36c37c2715b29ee with result from task 671 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/1512557d-3764-48c2-a56b-32d31de43b48_calibrate_947244_31.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 451d27fc72125a0fb603589aba416226 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b85f4ce1-e305-4764-bf9f-b5d65d438a0a_calibrate_947246_12.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b85f4ce1-e305-4764-bf9f-b5d65d438a0a_calibrate_947246_12.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6a31ae8a182439e11c0e4d78d390477 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ac460c04-aace-4026-b889-4db75566f317_calibrate_947246_57.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/ac460c04-aace-4026-b889-4db75566f317_calibrate_947246_57.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6dcf18f8c292581a31cd011f68882c0 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 55a466140e39871c1fd80e759756dc79 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf6923f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 1131 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/0ba2b8b2-606e-4fa7-a518-ab450ba76373_calibrate_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/0ba2b8b2-606e-4fa7-a518-ab450ba76373_calibrate_947140_41.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/3a39456f-e044-4868-a4d5-d524ffab480f_characterizeImage_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947140/3a39456f-e044-4868-a4d5-d524ffab480f_characterizeImage_947140_41.stderr 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f9f52a96a9c62df71f9ae9f22d33c25 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ac6ad650-6327-47de-80a7-0654fe27e581_calibrate_947248_29.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/ac6ad650-6327-47de-80a7-0654fe27e581_calibrate_947248_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:38 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e81f222df530ad906263613e39bc07d with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/37b6b1bd-b64a-4fe5-b373-824875a7cbc2_calibrate_947246_29.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/37b6b1bd-b64a-4fe5-b373-824875a7cbc2_calibrate_947246_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:39 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac13ea1a0d3aced85fd0d2c074d085db with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2d25828d-3597-4038-a012-831458119ffd_calibrate_947248_20.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/2d25828d-3597-4038-a012-831458119ffd_calibrate_947248_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:40 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17c242769aef7946ee4c9e067166e194 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash fdfbd37b51d93fa08d64d4fe237e13e4 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 1132 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/20b9473b-0898-4bd8-b17a-3dc599593502_characterizeImage_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/20b9473b-0898-4bd8-b17a-3dc599593502_characterizeImage_947144_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:41 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:42 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f260e262fc184a47b31a154d300ae9 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/14308b75-995f-4a47-8543-8ac23834bb97_calibrate_947244_12.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/14308b75-995f-4a47-8543-8ac23834bb97_calibrate_947244_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:43 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52068806b39f846eaf7cf1e0baf16da4 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/48c14552-2799-4747-be23-ead5984d9ff6_calibrate_947244_25.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/48c14552-2799-4747-be23-ead5984d9ff6_calibrate_947244_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:44 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:45 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:46 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:47 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:39:48 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbea3054f66b3458dbdbb65b0371267a with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash f278e64f9125dadb10abc99b5d3adc66 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf675fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 1133 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aebf3f3f-e783-45fc-bfbf-016cd6a03879_calibrate_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aebf3f3f-e783-45fc-bfbf-016cd6a03879_calibrate_947244_61.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/b8a80675-120d-4e01-beb1-4da743c00ad8_characterizeImage_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947244/b8a80675-120d-4e01-beb1-4da743c00ad8_characterizeImage_947244_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5348ff98fd7c6cd1ffb1abe7bdc83912 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f011e314-f0ef-4d37-9ed4-cf0f704bd72c_calibrate_947246_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/f011e314-f0ef-4d37-9ed4-cf0f704bd72c_calibrate_947246_49.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c115aa6b4e080a648d57e5a4be2ed663 with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4e7ae4a3-b641-491e-8972-c98dd0b2452f_calibrate_947246_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/4e7ae4a3-b641-491e-8972-c98dd0b2452f_calibrate_947246_37.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be81783b83f5bd200434d5747484136 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c33a4f2b-5688-4d1f-9d67-5cd9b1887c67_calibrate_947248_10.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/c33a4f2b-5688-4d1f-9d67-5cd9b1887c67_calibrate_947248_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d81147e54d0fcd3d3ecf259898db2a6 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/3bffb537-783e-448d-954d-b8bf40bf2920_calibrate_947248_25.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947248/3bffb537-783e-448d-954d-b8bf40bf2920_calibrate_947248_25.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a466140e39871c1fd80e759756dc79 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/0ba2b8b2-606e-4fa7-a518-ab450ba76373_calibrate_947140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947140/0ba2b8b2-606e-4fa7-a518-ab450ba76373_calibrate_947140_41.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69211dd6cc4002a9f2cef7c34fea5f5b with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d78b98db-0edc-42c1-b1f8-73f3de4358ec_calibrate_947142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/d78b98db-0edc-42c1-b1f8-73f3de4358ec_calibrate_947142_4.stderr 2024-03-14 05:40:08 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:09 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:10 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:11 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:12 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:13 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:14 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:15 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:16 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:17 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:18 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:19 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:19 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:19 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 656 try 0 failed parsl.dataflow.dflow INFO: Task 656 marked for retry parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash fdfbd37b51d93fa08d64d4fe237e13e4 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf68484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 1 launched on executor multi with executor id 1134 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5513129da84951d97039c76833a1e1e with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b043a2e1-064a-46a1-a67d-3e4d8aaff544_calibrate_947246_61.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947246/b043a2e1-064a-46a1-a67d-3e4d8aaff544_calibrate_947246_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:39 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:40 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:41 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:42 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:43 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:44 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:45 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:46 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:47 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:48 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:40:49 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 656 try 1 failed parsl.dataflow.dflow ERROR: Task 656 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fdfbd37b51d93fa08d64d4fe237e13e4 with result from task 656 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/86f3d590-d455-4b73-beb4-c1098731e48e_calibrate_947144_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f278e64f9125dadb10abc99b5d3adc66 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aebf3f3f-e783-45fc-bfbf-016cd6a03879_calibrate_947244_61.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947244/aebf3f3f-e783-45fc-bfbf-016cd6a03879_calibrate_947244_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:10 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:11 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:12 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:13 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:14 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:15 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:16 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:17 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:18 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:19 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:20 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:20 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:20 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:20 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:20 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:41:40 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:41:40 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:40 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:40 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:41 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:42 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:43 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:44 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:45 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:46 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:47 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:48 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:49 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:41:50 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:11 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:12 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:13 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:14 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:15 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:16 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:17 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:18 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:19 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:20 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:21 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:21 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:21 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:21 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:21 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:42:41 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:42:41 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:41 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:41 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:41 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:42 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:43 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:44 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:45 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:46 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:47 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:48 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:49 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:50 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:42:51 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d5662e1f021ba634a3f54ba662e942 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash ea76b824f6c0b63a1871b578ec636a99 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf69b22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 1135 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4fd608d5-8a97-4282-9832-4751c2f8c575_calibrate_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4fd608d5-8a97-4282-9832-4751c2f8c575_calibrate_947144_62.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ccdf97ec-c26b-400b-8d38-8032de484f4d_characterizeImage_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947144/ccdf97ec-c26b-400b-8d38-8032de484f4d_characterizeImage_947144_62.stderr 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:12 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:13 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:14 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:15 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:16 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:17 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:18 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:19 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:20 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:21 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:22 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:22 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:22 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:22 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:43:42 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:43:42 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:42 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:42 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:42 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:43 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:44 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea76b824f6c0b63a1871b578ec636a99 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4fd608d5-8a97-4282-9832-4751c2f8c575_calibrate_947144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947144/4fd608d5-8a97-4282-9832-4751c2f8c575_calibrate_947144_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:45 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:46 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:47 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:48 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:49 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:50 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:51 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:43:52 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:44:12 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:13 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:14 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:15 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:16 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:17 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:18 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:19 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:20 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:21 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:22 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:23 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:23 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18900f346a2aaf707a899c2394bb6ac5 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 40fd98d29f1825e1ea4ce585cf2b024b parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x148bf7ab0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 1136 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a88eb4cc-e55e-471f-9c97-f24fc7d476aa_calibrate_947142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a88eb4cc-e55e-471f-9c97-f24fc7d476aa_calibrate_947142_40.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9dd0598a-f882-4e4f-8e38-3c35a85bc954_characterizeImage_947142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/characterizeImage/947142/9dd0598a-f882-4e4f-8e38-3c35a85bc954_characterizeImage_947142_40.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:43 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:44 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:45 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031548,17031549,17031550,17031551,17031552,17031553,17031554,17031555,17031556,17031557,17031558,17031559,17031560,17031561,17031562,17031563,17031564,17031565,17031566,17031567,17031568,17031569,17031570,17031571,17031572,17031573,17031574,17031575,17031576,17031577' parsl.providers.slurm.slurm DEBUG: squeue returned 17031569 R 17031570 R 17031571 R 17031572 R 17031573 R 17031574 R 17031575 R 17031576 R 17031577 R 17031559 R 17031560 R 17031561 R 17031562 R 17031563 R 17031564 R 17031565 R 17031566 R 17031567 R 17031568 R 17031549 R 17031550 R 17031551 R 17031552 R 17031553 R 17031554 R 17031555 R 17031556 R 17031557 R 17031558 R 17031548 R parsl.providers.slurm.slurm DEBUG: Updating job 17031569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031570 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031576 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031559 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031560 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031561 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031562 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031563 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031549 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031550 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031551 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031552 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031553 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031554 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031555 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031556 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031557 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031558 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031548 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:46 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031578,17031579,17031580,17031581,17031582' parsl.providers.slurm.slurm DEBUG: squeue returned 17031578 R 17031579 R 17031580 R 17031581 R 17031582 R parsl.providers.slurm.slurm DEBUG: Updating job 17031578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031582 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:47 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:48 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031583,17031584,17031585,17031586,17031587,17031588,17031589,17031590,17031591,17031592,17031593,17031594,17031595,17031596,17031597,17031598,17031599,17031600,17031601,17031602' parsl.providers.slurm.slurm DEBUG: squeue returned 17031598 R 17031599 R 17031600 R 17031601 R 17031602 R 17031588 R 17031589 R 17031590 R 17031591 R 17031592 R 17031593 R 17031594 R 17031595 R 17031596 R 17031597 R 17031583 R 17031584 R 17031585 R 17031586 R 17031587 R parsl.providers.slurm.slurm DEBUG: Updating job 17031598 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031600 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031588 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031596 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031587 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:49 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:50 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:51 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:52 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING 2024-03-14 05:44:53 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17031603,17031604,17031605,17031606,17031607,17031608,17031609,17031610,17031611,17031612,17031613,17031614,17031615,17031616,17031617,17031618,17031619,17031620,17031621,17031622,17031623,17031624,17031625,17031626,17031627,17031628,17031629,17031630,17031631,17031632,17031633,17031634,17031635,17031636,17031637,17031638,17031639' parsl.providers.slurm.slurm DEBUG: squeue returned 17031605 R 17031604 R 17031639 R 17031629 R 17031630 R 17031631 R 17031632 R 17031633 R 17031634 R 17031635 R 17031636 R 17031637 R 17031638 R 17031619 R 17031620 R 17031621 R 17031622 R 17031623 R 17031624 R 17031625 R 17031626 R 17031627 R 17031628 R 17031610 R 17031611 R 17031612 R 17031613 R 17031614 R 17031615 R 17031616 R 17031617 R 17031618 R 17031603 R 17031606 R 17031607 R 17031608 R 17031609 R parsl.providers.slurm.slurm DEBUG: Updating job 17031605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031604 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031629 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031630 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031631 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031632 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031633 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031634 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031635 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031636 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031619 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031620 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031621 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031622 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031623 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031624 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031625 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031626 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031627 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031628 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031610 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031611 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031612 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031613 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031614 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031615 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031616 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031617 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031618 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031603 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031606 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031607 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031608 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17031609 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22591503910160 parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40fd98d29f1825e1ea4ce585cf2b024b with result from task 197 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1096 parsl.dataflow.dflow INFO: Tasks in state States.failed: 12 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 8 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a88eb4cc-e55e-471f-9c97-f24fc7d476aa_calibrate_947142_40.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/submit/DEEP/20201020/B1e/science#step1/20240314T053216Z/logs/calibrate/947142/a88eb4cc-e55e-471f-9c97-f24fc7d476aa_calibrate_947142_40.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7c03a22cb4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7c03a22cb4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0a14d822b53 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0a14d822b53 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8825612c4007 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8825612c4007 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a938f3949c98 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a938f3949c98 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28b97373e05c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28b97373e05c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66381487ac28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66381487ac28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 03eeb218f495 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 03eeb218f495 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ebe5fc088f86 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ebe5fc088f86 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7eb86c9fc2df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7eb86c9fc2df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5beca8c46241 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5beca8c46241 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f488aa1526c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f488aa1526c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b840870a7eb3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b840870a7eb3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a2a63a63a0c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a2a63a63a0c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00a97880df9e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00a97880df9e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af9f61364e27 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af9f61364e27 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 02a352b45b25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 02a352b45b25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae951496eaac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae951496eaac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fbab4ceb4c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fbab4ceb4c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4d19d7dad15 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4d19d7dad15 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fb888c530e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fb888c530e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d1ae29722f1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d1ae29722f1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b6496143bb8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b6496143bb8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf5ec340a9f4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf5ec340a9f4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 93358cd2b7bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 93358cd2b7bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 36eba3dba3ef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 36eba3dba3ef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47e53aead010 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47e53aead010 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d960d512d03 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d960d512d03 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 231c0a09d444 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 231c0a09d444 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f7c9b16646f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f7c9b16646f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 335d24a39dda parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 335d24a39dda parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd6fd6f83bf5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd6fd6f83bf5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc96a25317c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc96a25317c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1257a0c139b6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1257a0c139b6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4deea78ed52f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4deea78ed52f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75ec0c7949bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75ec0c7949bb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 111723558515 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 111723558515 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b38fc745f47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b38fc745f47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56ccce4f886c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56ccce4f886c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0fbb22af3e9b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0fbb22af3e9b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5092bbffe6b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5092bbffe6b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f815fcb3267 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f815fcb3267 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 982c55152697 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 982c55152697 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df4daa6ded06 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df4daa6ded06 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8cc30087fc8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8cc30087fc8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 882687269636 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 882687269636 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad74a59af4ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad74a59af4ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 230efdbf56b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 230efdbf56b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7759280d5db8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7759280d5db8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f07cbf964aff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f07cbf964aff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66d71ef42ad3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66d71ef42ad3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6032ca2ee66f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6032ca2ee66f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d81d9f768871 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d81d9f768871 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f97f483f2be5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f97f483f2be5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05558eb52b72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05558eb52b72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f0240b36d67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f0240b36d67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1487a03a3202 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1487a03a3202 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10c7c36f2386 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10c7c36f2386 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5dd3674d5632 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5dd3674d5632 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 325a19d4d538 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 325a19d4d538 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76d97c42a717 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76d97c42a717 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 441c211703b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 441c211703b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c18fa596510c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c18fa596510c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 877e32217b12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 877e32217b12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eee008ca6065 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eee008ca6065 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c7a38c10e5b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c7a38c10e5b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 31583014fa77 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 31583014fa77 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ad748077b89d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ad748077b89d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e59e374523d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e59e374523d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50f03431f9df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50f03431f9df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 934d65afd96a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 934d65afd96a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cbe2e03b562f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cbe2e03b562f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3e2b649c46b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3e2b649c46b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eec9ce199235 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eec9ce199235 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 257672e1cc3c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 257672e1cc3c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6995ecc93bb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6995ecc93bb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1acb93d0007d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1acb93d0007d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15cea2639244 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15cea2639244 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e4cde17de0c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e4cde17de0c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a60a29b16949 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a60a29b16949 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50c8a85d7ad9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50c8a85d7ad9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8285333652c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8285333652c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb7cb66cde46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb7cb66cde46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb05b8a09e5c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb05b8a09e5c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1221b9f39b9b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1221b9f39b9b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8313d626c302 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8313d626c302 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bdce7f488d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bdce7f488d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d6072707616 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d6072707616 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 657156d092eb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 657156d092eb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78d79f976c3f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78d79f976c3f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 680892135614 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 680892135614 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 350ac094c3c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 350ac094c3c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0630756a29f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0630756a29f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75e7f64a1563 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75e7f64a1563 2024-03-14 05:44:57 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:201 [INFO] cancelling 17031609 on provider checkpoint proc_lsst.multi INFO: cancelling 17031609 on provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:44:57 proc_lsst.multi:201 [INFO] cancelling 40354 on provider local proc_lsst.multi INFO: cancelling 40354 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 40354 2024-03-14 05:44:57 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:201 [INFO] cancelling 17031614 on provider checkpoint proc_lsst.multi INFO: cancelling 17031614 on provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:201 [INFO] cancelling 17031617 on provider checkpoint proc_lsst.multi INFO: cancelling 17031617 on provider checkpoint 2024-03-14 05:44:57 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 2024-03-14 05:44:57 proc_lsst.multi:201 [INFO] cancelling 17031602 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031602 on provider escience_a40 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031634 on provider checkpoint proc_lsst.multi INFO: cancelling 17031634 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031549 on provider astro proc_lsst.multi INFO: cancelling 17031549 on provider astro 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031637 on provider checkpoint proc_lsst.multi INFO: cancelling 17031637 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031607 on provider checkpoint proc_lsst.multi INFO: cancelling 17031607 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031613 on provider checkpoint proc_lsst.multi INFO: cancelling 17031613 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031589 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031589 on provider escience_a40 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031636 on provider checkpoint proc_lsst.multi INFO: cancelling 17031636 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031577 on provider astro proc_lsst.multi INFO: cancelling 17031577 on provider astro 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031553 on provider astro proc_lsst.multi INFO: cancelling 17031553 on provider astro 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031624 on provider checkpoint proc_lsst.multi INFO: cancelling 17031624 on provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint 2024-03-14 05:44:58 proc_lsst.multi:201 [INFO] cancelling 17031631 on provider checkpoint proc_lsst.multi INFO: cancelling 17031631 on provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031625 on provider checkpoint proc_lsst.multi INFO: cancelling 17031625 on provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031633 on provider checkpoint proc_lsst.multi INFO: cancelling 17031633 on provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031563 on provider astro proc_lsst.multi INFO: cancelling 17031563 on provider astro 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031550 on provider astro proc_lsst.multi INFO: cancelling 17031550 on provider astro 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031587 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031587 on provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031599 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031599 on provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031590 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031590 on provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031592 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031592 on provider escience_a40 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031632 on provider checkpoint proc_lsst.multi INFO: cancelling 17031632 on provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031575 on provider astro proc_lsst.multi INFO: cancelling 17031575 on provider astro 2024-03-14 05:44:59 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint 2024-03-14 05:44:59 proc_lsst.multi:201 [INFO] cancelling 17031608 on provider checkpoint proc_lsst.multi INFO: cancelling 17031608 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031579 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031579 on provider escience_rtx6k 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031585 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031585 on provider escience_a40 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031612 on provider checkpoint proc_lsst.multi INFO: cancelling 17031612 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031639 on provider checkpoint proc_lsst.multi INFO: cancelling 17031639 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031628 on provider checkpoint proc_lsst.multi INFO: cancelling 17031628 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031623 on provider checkpoint proc_lsst.multi INFO: cancelling 17031623 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031606 on provider checkpoint proc_lsst.multi INFO: cancelling 17031606 on provider checkpoint 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031557 on provider astro proc_lsst.multi INFO: cancelling 17031557 on provider astro 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031560 on provider astro proc_lsst.multi INFO: cancelling 17031560 on provider astro 2024-03-14 05:45:00 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro 2024-03-14 05:45:00 proc_lsst.multi:201 [INFO] cancelling 17031562 on provider astro proc_lsst.multi INFO: cancelling 17031562 on provider astro 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031555 on provider astro proc_lsst.multi INFO: cancelling 17031555 on provider astro 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031583 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031583 on provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031561 on provider astro proc_lsst.multi INFO: cancelling 17031561 on provider astro 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031622 on provider checkpoint proc_lsst.multi INFO: cancelling 17031622 on provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031616 on provider checkpoint proc_lsst.multi INFO: cancelling 17031616 on provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031598 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031598 on provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031594 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031594 on provider escience_a40 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031551 on provider astro proc_lsst.multi INFO: cancelling 17031551 on provider astro 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031629 on provider checkpoint proc_lsst.multi INFO: cancelling 17031629 on provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031630 on provider checkpoint proc_lsst.multi INFO: cancelling 17031630 on provider checkpoint 2024-03-14 05:45:01 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro 2024-03-14 05:45:01 proc_lsst.multi:201 [INFO] cancelling 17031573 on provider astro proc_lsst.multi INFO: cancelling 17031573 on provider astro 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031605 on provider checkpoint proc_lsst.multi INFO: cancelling 17031605 on provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031559 on provider astro proc_lsst.multi INFO: cancelling 17031559 on provider astro 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031578 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031578 on provider escience_rtx6k 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031603 on provider checkpoint proc_lsst.multi INFO: cancelling 17031603 on provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031611 on provider checkpoint proc_lsst.multi INFO: cancelling 17031611 on provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031597 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031597 on provider escience_a40 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031621 on provider checkpoint proc_lsst.multi INFO: cancelling 17031621 on provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031610 on provider checkpoint proc_lsst.multi INFO: cancelling 17031610 on provider checkpoint 2024-03-14 05:45:02 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k 2024-03-14 05:45:02 proc_lsst.multi:201 [INFO] cancelling 17031582 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031582 on provider escience_rtx6k 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031596 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031596 on provider escience_a40 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031576 on provider astro proc_lsst.multi INFO: cancelling 17031576 on provider astro 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031566 on provider astro proc_lsst.multi INFO: cancelling 17031566 on provider astro 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031600 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031600 on provider escience_a40 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031635 on provider checkpoint proc_lsst.multi INFO: cancelling 17031635 on provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031618 on provider checkpoint proc_lsst.multi INFO: cancelling 17031618 on provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031620 on provider checkpoint proc_lsst.multi INFO: cancelling 17031620 on provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031638 on provider checkpoint proc_lsst.multi INFO: cancelling 17031638 on provider checkpoint 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031556 on provider astro proc_lsst.multi INFO: cancelling 17031556 on provider astro 2024-03-14 05:45:03 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro 2024-03-14 05:45:03 proc_lsst.multi:201 [INFO] cancelling 17031570 on provider astro proc_lsst.multi INFO: cancelling 17031570 on provider astro 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031619 on provider checkpoint proc_lsst.multi INFO: cancelling 17031619 on provider checkpoint 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031626 on provider checkpoint proc_lsst.multi INFO: cancelling 17031626 on provider checkpoint 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031595 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031595 on provider escience_a40 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031586 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031586 on provider escience_a40 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031552 on provider astro proc_lsst.multi INFO: cancelling 17031552 on provider astro 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031554 on provider astro proc_lsst.multi INFO: cancelling 17031554 on provider astro 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031564 on provider astro proc_lsst.multi INFO: cancelling 17031564 on provider astro 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031581 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031581 on provider escience_rtx6k 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031572 on provider astro proc_lsst.multi INFO: cancelling 17031572 on provider astro 2024-03-14 05:45:04 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro 2024-03-14 05:45:04 proc_lsst.multi:201 [INFO] cancelling 17031574 on provider astro proc_lsst.multi INFO: cancelling 17031574 on provider astro 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031604 on provider checkpoint proc_lsst.multi INFO: cancelling 17031604 on provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031568 on provider astro proc_lsst.multi INFO: cancelling 17031568 on provider astro 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031580 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031580 on provider escience_rtx6k 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031567 on provider astro proc_lsst.multi INFO: cancelling 17031567 on provider astro 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031571 on provider astro proc_lsst.multi INFO: cancelling 17031571 on provider astro 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031588 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031588 on provider escience_a40 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031569 on provider astro proc_lsst.multi INFO: cancelling 17031569 on provider astro 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031615 on provider checkpoint proc_lsst.multi INFO: cancelling 17031615 on provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031627 on provider checkpoint proc_lsst.multi INFO: cancelling 17031627 on provider checkpoint 2024-03-14 05:45:05 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 2024-03-14 05:45:05 proc_lsst.multi:201 [INFO] cancelling 17031593 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031593 on provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031548 on provider astro proc_lsst.multi INFO: cancelling 17031548 on provider astro 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031558 on provider astro proc_lsst.multi INFO: cancelling 17031558 on provider astro 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031601 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031601 on provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031565 on provider astro proc_lsst.multi INFO: cancelling 17031565 on provider astro 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031584 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031584 on provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031591 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031591 on provider escience_a40 parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-14 05:45:06 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-14 05:45:06 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17031603', '17031604', '17031605', '17031606', '17031607', '17031608', '17031609', '17031610', '17031611', '17031612', '17031613', '17031614', '17031615', '17031616', '17031617', '17031618', '17031619', '17031620', '17031621', '17031622', '17031623', '17031624', '17031625', '17031626', '17031627', '17031628', '17031629', '17031630', '17031631', '17031632', '17031633', '17031634', '17031635', '17031636', '17031637', '17031638', '17031639', '17031548', '17031549', '17031550', '17031551', '17031552', '17031553', '17031554', '17031555', '17031556', '17031557', '17031558', '17031559', '17031560', '17031561', '17031562', '17031563', '17031564', '17031565', '17031566', '17031567', '17031568', '17031569', '17031570', '17031571', '17031572', '17031573', '17031574', '17031575', '17031576', '17031577', '17031583', '17031584', '17031585', '17031586', '17031587', '17031588', '17031589', '17031590', '17031591', '17031592', '17031593', '17031594', '17031595', '17031596', '17031597', '17031598', '17031599', '17031600', '17031601', '17031602', '17031578', '17031579', '17031580', '17031581', '17031582', '40354'] proc_lsst.multi INFO: new jobs since last cancel ['17031603', '17031604', '17031605', '17031606', '17031607', '17031608', '17031609', '17031610', '17031611', '17031612', '17031613', '17031614', '17031615', '17031616', '17031617', '17031618', '17031619', '17031620', '17031621', '17031622', '17031623', '17031624', '17031625', '17031626', '17031627', '17031628', '17031629', '17031630', '17031631', '17031632', '17031633', '17031634', '17031635', '17031636', '17031637', '17031638', '17031639', '17031548', '17031549', '17031550', '17031551', '17031552', '17031553', '17031554', '17031555', '17031556', '17031557', '17031558', '17031559', '17031560', '17031561', '17031562', '17031563', '17031564', '17031565', '17031566', '17031567', '17031568', '17031569', '17031570', '17031571', '17031572', '17031573', '17031574', '17031575', '17031576', '17031577', '17031583', '17031584', '17031585', '17031586', '17031587', '17031588', '17031589', '17031590', '17031591', '17031592', '17031593', '17031594', '17031595', '17031596', '17031597', '17031598', '17031599', '17031600', '17031601', '17031602', '17031578', '17031579', '17031580', '17031581', '17031582', '40354'] 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031603 in provider checkpoint proc_lsst.multi INFO: found job 17031603 in provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031603 on provider checkpoint proc_lsst.multi INFO: cancelling 17031603 on provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031604 in provider checkpoint proc_lsst.multi INFO: found job 17031604 in provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031604 on provider checkpoint proc_lsst.multi INFO: cancelling 17031604 on provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031605 in provider checkpoint proc_lsst.multi INFO: found job 17031605 in provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031605 on provider checkpoint proc_lsst.multi INFO: cancelling 17031605 on provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:146 [INFO] found job 17031606 in provider checkpoint proc_lsst.multi INFO: found job 17031606 in provider checkpoint 2024-03-14 05:45:06 proc_lsst.multi:201 [INFO] cancelling 17031606 on provider checkpoint proc_lsst.multi INFO: cancelling 17031606 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031607 in provider checkpoint proc_lsst.multi INFO: found job 17031607 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031607 on provider checkpoint proc_lsst.multi INFO: cancelling 17031607 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031608 in provider checkpoint proc_lsst.multi INFO: found job 17031608 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031608 on provider checkpoint proc_lsst.multi INFO: cancelling 17031608 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031609 in provider checkpoint proc_lsst.multi INFO: found job 17031609 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031609 on provider checkpoint proc_lsst.multi INFO: cancelling 17031609 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031610 in provider checkpoint proc_lsst.multi INFO: found job 17031610 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031610 on provider checkpoint proc_lsst.multi INFO: cancelling 17031610 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031611 in provider checkpoint proc_lsst.multi INFO: found job 17031611 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031611 on provider checkpoint proc_lsst.multi INFO: cancelling 17031611 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031612 in provider checkpoint proc_lsst.multi INFO: found job 17031612 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031612 on provider checkpoint proc_lsst.multi INFO: cancelling 17031612 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031613 in provider checkpoint proc_lsst.multi INFO: found job 17031613 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031613 on provider checkpoint proc_lsst.multi INFO: cancelling 17031613 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031614 in provider checkpoint proc_lsst.multi INFO: found job 17031614 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031614 on provider checkpoint proc_lsst.multi INFO: cancelling 17031614 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031615 in provider checkpoint proc_lsst.multi INFO: found job 17031615 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031615 on provider checkpoint proc_lsst.multi INFO: cancelling 17031615 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031616 in provider checkpoint proc_lsst.multi INFO: found job 17031616 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031616 on provider checkpoint proc_lsst.multi INFO: cancelling 17031616 on provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:146 [INFO] found job 17031617 in provider checkpoint proc_lsst.multi INFO: found job 17031617 in provider checkpoint 2024-03-14 05:45:07 proc_lsst.multi:201 [INFO] cancelling 17031617 on provider checkpoint proc_lsst.multi INFO: cancelling 17031617 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031618 in provider checkpoint proc_lsst.multi INFO: found job 17031618 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031618 on provider checkpoint proc_lsst.multi INFO: cancelling 17031618 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031619 in provider checkpoint proc_lsst.multi INFO: found job 17031619 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031619 on provider checkpoint proc_lsst.multi INFO: cancelling 17031619 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031620 in provider checkpoint proc_lsst.multi INFO: found job 17031620 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031620 on provider checkpoint proc_lsst.multi INFO: cancelling 17031620 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031621 in provider checkpoint proc_lsst.multi INFO: found job 17031621 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031621 on provider checkpoint proc_lsst.multi INFO: cancelling 17031621 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031622 in provider checkpoint proc_lsst.multi INFO: found job 17031622 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031622 on provider checkpoint proc_lsst.multi INFO: cancelling 17031622 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031623 in provider checkpoint proc_lsst.multi INFO: found job 17031623 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031623 on provider checkpoint proc_lsst.multi INFO: cancelling 17031623 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031624 in provider checkpoint proc_lsst.multi INFO: found job 17031624 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031624 on provider checkpoint proc_lsst.multi INFO: cancelling 17031624 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031625 in provider checkpoint proc_lsst.multi INFO: found job 17031625 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031625 on provider checkpoint proc_lsst.multi INFO: cancelling 17031625 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031626 in provider checkpoint proc_lsst.multi INFO: found job 17031626 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031626 on provider checkpoint proc_lsst.multi INFO: cancelling 17031626 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031627 in provider checkpoint proc_lsst.multi INFO: found job 17031627 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031627 on provider checkpoint proc_lsst.multi INFO: cancelling 17031627 on provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:146 [INFO] found job 17031628 in provider checkpoint proc_lsst.multi INFO: found job 17031628 in provider checkpoint 2024-03-14 05:45:08 proc_lsst.multi:201 [INFO] cancelling 17031628 on provider checkpoint proc_lsst.multi INFO: cancelling 17031628 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031629 in provider checkpoint proc_lsst.multi INFO: found job 17031629 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031629 on provider checkpoint proc_lsst.multi INFO: cancelling 17031629 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031630 in provider checkpoint proc_lsst.multi INFO: found job 17031630 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031630 on provider checkpoint proc_lsst.multi INFO: cancelling 17031630 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031631 in provider checkpoint proc_lsst.multi INFO: found job 17031631 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031631 on provider checkpoint proc_lsst.multi INFO: cancelling 17031631 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031632 in provider checkpoint proc_lsst.multi INFO: found job 17031632 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031632 on provider checkpoint proc_lsst.multi INFO: cancelling 17031632 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031633 in provider checkpoint proc_lsst.multi INFO: found job 17031633 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031633 on provider checkpoint proc_lsst.multi INFO: cancelling 17031633 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031634 in provider checkpoint proc_lsst.multi INFO: found job 17031634 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031634 on provider checkpoint proc_lsst.multi INFO: cancelling 17031634 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031635 in provider checkpoint proc_lsst.multi INFO: found job 17031635 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031635 on provider checkpoint proc_lsst.multi INFO: cancelling 17031635 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031636 in provider checkpoint proc_lsst.multi INFO: found job 17031636 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031636 on provider checkpoint proc_lsst.multi INFO: cancelling 17031636 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031637 in provider checkpoint proc_lsst.multi INFO: found job 17031637 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031637 on provider checkpoint proc_lsst.multi INFO: cancelling 17031637 on provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:146 [INFO] found job 17031638 in provider checkpoint proc_lsst.multi INFO: found job 17031638 in provider checkpoint 2024-03-14 05:45:09 proc_lsst.multi:201 [INFO] cancelling 17031638 on provider checkpoint proc_lsst.multi INFO: cancelling 17031638 on provider checkpoint 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031639 in provider checkpoint proc_lsst.multi INFO: found job 17031639 in provider checkpoint 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031639 on provider checkpoint proc_lsst.multi INFO: cancelling 17031639 on provider checkpoint 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031548 in provider astro proc_lsst.multi INFO: found job 17031548 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031548 on provider astro proc_lsst.multi INFO: cancelling 17031548 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031549 in provider astro proc_lsst.multi INFO: found job 17031549 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031549 on provider astro proc_lsst.multi INFO: cancelling 17031549 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031550 in provider astro proc_lsst.multi INFO: found job 17031550 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031550 on provider astro proc_lsst.multi INFO: cancelling 17031550 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031551 in provider astro proc_lsst.multi INFO: found job 17031551 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031551 on provider astro proc_lsst.multi INFO: cancelling 17031551 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031552 in provider astro proc_lsst.multi INFO: found job 17031552 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031552 on provider astro proc_lsst.multi INFO: cancelling 17031552 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031553 in provider astro proc_lsst.multi INFO: found job 17031553 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031553 on provider astro proc_lsst.multi INFO: cancelling 17031553 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031554 in provider astro proc_lsst.multi INFO: found job 17031554 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031554 on provider astro proc_lsst.multi INFO: cancelling 17031554 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031555 in provider astro proc_lsst.multi INFO: found job 17031555 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031555 on provider astro proc_lsst.multi INFO: cancelling 17031555 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031556 in provider astro proc_lsst.multi INFO: found job 17031556 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031556 on provider astro proc_lsst.multi INFO: cancelling 17031556 on provider astro 2024-03-14 05:45:10 proc_lsst.multi:146 [INFO] found job 17031557 in provider astro proc_lsst.multi INFO: found job 17031557 in provider astro 2024-03-14 05:45:10 proc_lsst.multi:201 [INFO] cancelling 17031557 on provider astro proc_lsst.multi INFO: cancelling 17031557 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031558 in provider astro proc_lsst.multi INFO: found job 17031558 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031558 on provider astro proc_lsst.multi INFO: cancelling 17031558 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031559 in provider astro proc_lsst.multi INFO: found job 17031559 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031559 on provider astro proc_lsst.multi INFO: cancelling 17031559 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031560 in provider astro proc_lsst.multi INFO: found job 17031560 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031560 on provider astro proc_lsst.multi INFO: cancelling 17031560 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031561 in provider astro proc_lsst.multi INFO: found job 17031561 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031561 on provider astro proc_lsst.multi INFO: cancelling 17031561 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031562 in provider astro proc_lsst.multi INFO: found job 17031562 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031562 on provider astro proc_lsst.multi INFO: cancelling 17031562 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031563 in provider astro proc_lsst.multi INFO: found job 17031563 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031563 on provider astro proc_lsst.multi INFO: cancelling 17031563 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031564 in provider astro proc_lsst.multi INFO: found job 17031564 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031564 on provider astro proc_lsst.multi INFO: cancelling 17031564 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031565 in provider astro proc_lsst.multi INFO: found job 17031565 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031565 on provider astro proc_lsst.multi INFO: cancelling 17031565 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031566 in provider astro proc_lsst.multi INFO: found job 17031566 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031566 on provider astro proc_lsst.multi INFO: cancelling 17031566 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031567 in provider astro proc_lsst.multi INFO: found job 17031567 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031567 on provider astro proc_lsst.multi INFO: cancelling 17031567 on provider astro 2024-03-14 05:45:11 proc_lsst.multi:146 [INFO] found job 17031568 in provider astro proc_lsst.multi INFO: found job 17031568 in provider astro 2024-03-14 05:45:11 proc_lsst.multi:201 [INFO] cancelling 17031568 on provider astro proc_lsst.multi INFO: cancelling 17031568 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031569 in provider astro proc_lsst.multi INFO: found job 17031569 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031569 on provider astro proc_lsst.multi INFO: cancelling 17031569 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031570 in provider astro proc_lsst.multi INFO: found job 17031570 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031570 on provider astro proc_lsst.multi INFO: cancelling 17031570 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031571 in provider astro proc_lsst.multi INFO: found job 17031571 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031571 on provider astro proc_lsst.multi INFO: cancelling 17031571 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031572 in provider astro proc_lsst.multi INFO: found job 17031572 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031572 on provider astro proc_lsst.multi INFO: cancelling 17031572 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031573 in provider astro proc_lsst.multi INFO: found job 17031573 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031573 on provider astro proc_lsst.multi INFO: cancelling 17031573 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031574 in provider astro proc_lsst.multi INFO: found job 17031574 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031574 on provider astro proc_lsst.multi INFO: cancelling 17031574 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031575 in provider astro proc_lsst.multi INFO: found job 17031575 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031575 on provider astro proc_lsst.multi INFO: cancelling 17031575 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031576 in provider astro proc_lsst.multi INFO: found job 17031576 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031576 on provider astro proc_lsst.multi INFO: cancelling 17031576 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031577 in provider astro proc_lsst.multi INFO: found job 17031577 in provider astro 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031577 on provider astro proc_lsst.multi INFO: cancelling 17031577 on provider astro 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031583 in provider escience_a40 proc_lsst.multi INFO: found job 17031583 in provider escience_a40 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031583 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031583 on provider escience_a40 2024-03-14 05:45:12 proc_lsst.multi:146 [INFO] found job 17031584 in provider escience_a40 proc_lsst.multi INFO: found job 17031584 in provider escience_a40 2024-03-14 05:45:12 proc_lsst.multi:201 [INFO] cancelling 17031584 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031584 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031585 in provider escience_a40 proc_lsst.multi INFO: found job 17031585 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031585 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031585 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031586 in provider escience_a40 proc_lsst.multi INFO: found job 17031586 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031586 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031586 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031587 in provider escience_a40 proc_lsst.multi INFO: found job 17031587 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031587 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031587 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031588 in provider escience_a40 proc_lsst.multi INFO: found job 17031588 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031588 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031588 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031589 in provider escience_a40 proc_lsst.multi INFO: found job 17031589 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031589 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031589 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031590 in provider escience_a40 proc_lsst.multi INFO: found job 17031590 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031590 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031590 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031591 in provider escience_a40 proc_lsst.multi INFO: found job 17031591 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031591 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031591 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031592 in provider escience_a40 proc_lsst.multi INFO: found job 17031592 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031592 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031592 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031593 in provider escience_a40 proc_lsst.multi INFO: found job 17031593 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031593 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031593 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031594 in provider escience_a40 proc_lsst.multi INFO: found job 17031594 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031594 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031594 on provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:146 [INFO] found job 17031595 in provider escience_a40 proc_lsst.multi INFO: found job 17031595 in provider escience_a40 2024-03-14 05:45:13 proc_lsst.multi:201 [INFO] cancelling 17031595 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031595 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031596 in provider escience_a40 proc_lsst.multi INFO: found job 17031596 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031596 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031596 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031597 in provider escience_a40 proc_lsst.multi INFO: found job 17031597 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031597 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031597 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031598 in provider escience_a40 proc_lsst.multi INFO: found job 17031598 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031598 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031598 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031599 in provider escience_a40 proc_lsst.multi INFO: found job 17031599 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031599 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031599 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031600 in provider escience_a40 proc_lsst.multi INFO: found job 17031600 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031600 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031600 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031601 in provider escience_a40 proc_lsst.multi INFO: found job 17031601 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031601 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031601 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031602 in provider escience_a40 proc_lsst.multi INFO: found job 17031602 in provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031602 on provider escience_a40 proc_lsst.multi INFO: cancelling 17031602 on provider escience_a40 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031578 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031578 in provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031578 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031578 on provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031579 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031579 in provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031579 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031579 on provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031580 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031580 in provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031580 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031580 on provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:146 [INFO] found job 17031581 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031581 in provider escience_rtx6k 2024-03-14 05:45:14 proc_lsst.multi:201 [INFO] cancelling 17031581 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031581 on provider escience_rtx6k 2024-03-14 05:45:15 proc_lsst.multi:146 [INFO] found job 17031582 in provider escience_rtx6k proc_lsst.multi INFO: found job 17031582 in provider escience_rtx6k 2024-03-14 05:45:15 proc_lsst.multi:201 [INFO] cancelling 17031582 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17031582 on provider escience_rtx6k 2024-03-14 05:45:15 proc_lsst.multi:146 [INFO] found job 40354 in provider local proc_lsst.multi INFO: found job 40354 in provider local 2024-03-14 05:45:15 proc_lsst.multi:201 [INFO] cancelling 40354 on provider local proc_lsst.multi INFO: cancelling 40354 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 40354 parsl.providers.local.local WARNING: Failed to kill PID: 40354 and child processes on local 2024-03-14 05:45:16 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 754.8486 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201020_B1e_science#step1_20240314T053216Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 754.8552 seconds; current memory usage: 0.392 Gibyte, delta: 0.013 Gibyte, peak delta: 0.013 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 775.9617 seconds; current memory usage: 0.392 Gibyte, delta: 0.255 Gibyte, peak delta: 0.255 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.392 Gibyte (main), 0.392 Gibyte (largest child process) Run Id: None Run Name: DEEP_20201020_B1e_science#step1_20240314T053216Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 12m58.361s user 2m27.987s sys 1m24.380s